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Section  A  —  Introduction  to  the  A-P-A  T&E  Process  Manual 

1.1.  (General.  This  Air  Force  Manual  (AFMAN)  implements  the  Air  Force  (AF)  test  and  evaluation 
(T&E)  process  for  the  Airframe  Propulsion  and  Avionics  (A-P-A)  mission  area  systems  for  aircraft 
(manned  or  unmanned)  and  cruise  missiles.  It  is  intended  to  be  universal  for  use  by  both  the  government 
and  contractors  during  any  phase  of  the  acquisition  cycle  in  a  wide  variety  of  environments,  including 
government  owned/operated  test  facilities,  commercial  labs,  large  open  air  ranges  (OARs),  Air  Force  Test 
Centers,  Product  Centers,  Air  Logistics  Centers  (ALCs),  Air  Force  Operational  Test  and  Evaluation  Cen¬ 
ter  (AFOTEC),  and  major  air  commands.  It  is  a  guide  for  use  by  program  managers,  test  managers,  test 
engineers,  test  organization  personnel,  major  command  headquarters  staffs,  and  others  involved  in  T&E 
of  A-P-A  mission  area  systems.  This  Air  Force  Manual  (AFMAN)  takes  the  Air  Force  T&E  process 
described  in  Air  Force  Instruction  (AFI)  99-103,  Air  Force  Test  and  Evaluation  Process,  and  tailors  it  for 
A-P-A  mission  area  systems.  This  manual,  as  well  as  AFI  99-103  and  its  associated  instructions  and  man¬ 
uals  has  application  for  test  items  made  in  the  United  States  or  in  foreign  countries  and  applies  to  Devel¬ 
opmental  T&E  (DT&E),  Operational’  (OT&E),  Qualification’  (QT&E),  Follow-on  Operational 
’(FOT&E),  Qualification  Opera-tional  ’(QOT&E),  and  Initial  Operational  T&E  (lOT&E). 

1.1.1.  This  manual  was  written  assuming  that  A-P-A  mission  area  systems  are  part  of  a  larger  system, 
the  weapon  system  which  is  defined  as  "A  composite  of  prime  operating  resources  and  equipment 
including  aircraft,  weapons,  weapon  racks/launchers,  personnel,  software,  etc.,  and  ancillary 
resources  and  equipment  including  training  and  support  equipment,  logistics,  etc.,  used  directly  by  the 
armed  forces  to  provide  an  operational  combat  capability."  As  such,  the  A-P-A  mission  area  T&E 
may  require  the  integration,  use,  and  support  of  non-A-P-A  mission  area  systems  to  accomplish 
A-P-A  system  test  objectives.  Other  process  manuals  relating  to  T&E  of  aircraft  systems  include 
Electronic  Warfare  (EW);  Command,  Control,  Communication,  Computers,  and  Intelligence  (C4I); 
Space;  and  Armament/Munitions  and  are  contained  in  AFMANs  99-112, 99-111, 99-113,  and  99-104, 
respectively.  See  Attachment  3  for  additional  T&E  related  references. 

1.2.  Scope.  This  manual  covers  A-P-A  mission  areas  and  systems  associated  with  fixed  and  rotary  wing 
aircraft,  unmanned  aerial  vehicles  (UAV),  and  cruise  missiles.  For  these  air  vehicles,  related  systems  and 
disciplines  covered  in  this  A-P-A  manual  are: 

•  Airframe.  Flying  qualities,  flight  controls,  structures  (airframe  itself,  inlet  design,  loads,  and 
flutter),  performance,  stores  integration  and  separation  characteristics,  installed  engine  perfor¬ 
mance,  mechanical  systems  (i.e.,  hydraulic,  electrical,  environmental  control,  landing  gear),  crew 
stations  (cockpit  layout,  payload  bay,  etc.),  and  mission/auxiliary  systems  (i.e.,  airframe  mounted 
auxiliary  drives/gearbox,  generators.). 

•  Propulsion.  Powerplant,  inlet  effects,  nozzle,  uninstalled  engine  performance,  and  auxiliary 
power  including  systems  such  as  jet  fuel  starter  (JFS)  and  auxiliary  power  unit(APU). 

•  Avionics.  Sensors,  communications,  navigation,  and  identification  (CNI),  guidance,  electronic 
warfare  (EW)  integration,  support/mission  systems  [includes  stores  management  systems,  fire 
control  systems,  weapon  integration,  air-to-air  (A/A)  and  air-to-ground  (A/G)  delivery  accuracy, 
controls  and  displays,  mission  operational  flight  programs  (OFPS),  etc.]. 
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1.2.1.  Although  other  necessary  specialty  disciplines  such  as:  aerial  delivery,  logistics,  human  fac¬ 
tors,  environmental/climatic,  instrumentation,  software,  and  safety  are  not  specifically  called  out  in 
the  A-P-A  mission  area  discussions,  they  are  common  and  of  paramount  importance  to  these  func¬ 
tional  mission  areas.  Their  involvement  in  the  test  process  is  addressed  within  the  templates  in  section 
3.2  of  this  manual.  Also,  because  most  A-P-A  system,  subsystem,  and  component  hardware  is  con¬ 
trolled  by  software,  A-P-A  related  embedded  software  T&E  is  also  covered  in  section  3.2  of  this  man¬ 
ual.  Details  of  software  only  and  hardware  only  T&E  are  not  included  within  this  manual,  instead, 
software  and  hardware  are  treated  as  merged  elements  of  functional  units  (i.e.,  components,  sub¬ 
systems  and  systems). 

1.2.2.  A-P-A  systems  testing  that  requires  the  use  of  stores,  missiles,  or  munitions  represent  air  vehi¬ 
cle/weapon  integration  tests.  Air  vehicle/weapon  integration  testing  is  common  to  both  the  A-P-A 
T&E  process  and  the  Armament/Munitions  T&E  process.  Generally,  if  the  air  vehicle  is  in  develop¬ 
ment  or  being  modified,  integration  of  inventory  armament/munitions  is  tested  via  the  A-P-A  T&E 
process.  If  the  armament/munitions  is  in  develop-ment  or  being  modified  and  being  certified  for  car¬ 
riage  on  a  mature  platform,  the  system  will  generally  be  tested  via  the  Armament/Munitions  T&E  pro¬ 
cess.  Integration  of  new  munitions  onto  a  new  air  vehicle  will  be  tested  using  a  combination  of  the 
A-P-A  and  Armament/Munitions  test  processes. 

1.3.  Document  Roadmap.  This  manual  is  divided  into  the  three  separate  and  distinct  areas  shown  in 
Figure  1.1.  The  first  area,  which  contains  sections  1  and  2,  is  in  narrative  format  and  introduces  the 
A-P-A  T&E  processes  at  a  high  level.  The  second  area,  section  3,  addresses  the  application  of  the  A-P-A 
T&E  process  which  uses  a  work  breakdown  structure  (WBS),  combined  with  a  template  format  (similar 
to  Willoughby  Templates,  Department  Of  Defense  (DOD)  4245. 7-M,  Transition  From  Development  to 
Production)  to  subdivide  the  A-P-A  mission  systems/disciplines  into  finer  details.  The  templateAVBS 
format  allows  the  reader  to  quickly  gather  necessary  information  at  a  level  of  detail  controlled  by  the 
reader  and  can  easily  accommodate  changes,  updates,  and  more  detailed  levels.  The  third  and  final  area 
is  comprised  of  the  attachments. 


AFMAN99-110  3  JULY  1995 
Figure  1.1.  A-P-A  Manual  Roadmap. 


7 


Sections  1&2 

-  Nairative  Forniat 

-  Common  to  all  A-P-A  Mission  Areas 


Section  3 

•  Temdate  Fomiat 
-  A-P-A  T&E  Pi’ocess  Appiicali on 


-  Provides  Infoimation  on: 

-  General  overview  (Sec.  1) 

-  A-P-A  T&E  Process  (Sec.2) 


Uses  WBS  to  Breakdowii  APA 
into  Details 

Provides  Details  of: 

-  Test  Process 

-  Mission  Req/Integ. 

-  Documents 

-  Embedded  Software 

-  Exit  Criteria 

-  Resources 


Attaclinients 


-  Narrative  Format 


1.  Aaon^’ms,  Abbreviations, 
Bibliography,  and  Glossat^^  of 
Terms 

2.  ^"stem  Development 

3.  T&E  Resources 


1.4.  Need.  The  need  for  implementing  a  disciplined  Air  Force  T&E  process  is  more  critical  now  than 
ever  before.  The  complexity  of  today’s  highly  integrated  systems  demand  the  implementation  of  a  pro¬ 
cess  that  will  enable  the  efficient  use  of  limited  resources,  minimize  risk  during  weapon  system  develop¬ 
ment  and  provide  an  effective,  quality  product  that  will  satisfy  user  requirements.  The  A-P-A  T&E 
process  described  in  this  manual  has  been  developed  to  help  guide  you  through  the  steps  necessary  to  plan 
and  execute  your  test  effort.  By  giving  proper  emphasis  to  pre-test  planning  and  post-test  evaluation,  you 
will  minimize  the  risks  and  costs  hidden  within  weapon  system  development  and  identify  deficiencies 
before  the  system  is  operationally  tested  and  fielded.  Early  involvement  of  test  centers  helps  to  reduce 
cost  and  improves  the  timeliness  and  effectiveness  of  testing.  The  manual  is  written  to  provide  you,  the 
test  planner,  guidance  on  what  to  do  to  effectively  implement  the  T&E  process,  not  how  to  do  it.  "How 
to"  details  are  contained  in  a  variety  of  published  guides  and  handbooks  located  at  test  centers,  program 
offices,  laboratories,  etc. 

1.4.1.  This  manual  implements  the  Air  Force  T&E  process  for  A-P-A  mission  area  systems,  and  cov¬ 
ers  items  from  the  operational  inventory,  developmental  or  research  items,  as  well  as  concept  evalua¬ 
tions.  The  manual  is  a  specific  application  of  the  generic  Air  Force  T&E  process  described  in  API 
99-103  and  covers  a  wide  breadth  of  airborne  weapon  system  functionalities  and  specialties,  including 
embedded  software  which  supports  A-P-A  systems.  It  is  tailored  towards  personnel  not  intimately 
familiar  with  A-P-A  related  T&E  or  how  T&E  specifically  fits  into  the  Department  Of  Defense 
(DOD)  acquisition  process  (DOD  5000  series).  This  includes:  experienced  acquisition  managers  with 
limited  T&E  experience,  inexperienced  acquisition  managers  and  personnel,  inexperienced  testers 
(both  within  and  not  within  A-P-A  related  mission  area),  and  personnel  not  intimately  famiUar  with 
the  acquisition  process.  Experienced  testers  may  also  find  this  manual  useful  as  an  A-P-A  T&E  doc¬ 
umentation  source. 

1.5.  Direction.  HQ  USAF/TE  defines  and  directs  the  use  of  the  Air  Force  T&E  process.  The  A-P-A 
Single-Face-To-Customer  (SFTC)  office  is  responsible  for  documenting  and  advocating  the  A-P-A  T&E 
process.  Stakeholders  in  the  A-P-A  T&E  process  include,  but  are  not  limited  to:  Air  Force  Materiel 
Command  (AFMC)/DO/XR/EN/LG,  AFOTEC,  Responsible  Test  Organizations  (RTOs),  Participating 
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Test  Organizations  (PTOs),  Operational  Test  Agencies  (OTAs),  Aeronautical  Systems  Center  (ASC), 
System  Program  Offices  (SPOs),  Air  Logistics  Center  (ALC),  Electronic  Systems  Center  (ESC),  Space 
and  Missile  Center  (SMC),  Laboratories,  and  Major  Range  and  Test  Facility  Bases  (MRTFBs). 

1.6.  A-P-A  Single-Face-To-Customer  Office.  The  goal  of  the  A-P-A  SFTC  office  is  to  improve  the  effi¬ 
ciency  and  cost  effectiveness  of  A-P-A  T&E  by  assisting  customers  in  early  application  of  the  A-P-A 
T&E  process,  identifying  risks  in  test  options  available  to  the  customer,  and  helping  the  customer  under¬ 
stand  the  capabilities  and  test  applications  of  the  T&E  resources  available  to  them.  The  SFTC  Office 
brings  in  the  expertise  of  the  above  mentioned  stakeholders  so  that  the  customer  understands  the  capabil¬ 
ities  of  the  available  T&E  resources.  The  objective  is  to  reduce  the  programmatic  risks  to  performance, 
cost,  and  schedule.  The  interfaces  used  by  the  SFTC  offices  to  provide  services  are  presented  in  Figure 
1.2.  and  also  in  the  SFTC  Mission  Needs  Statement  (MNS) 

Figure  1.2.  T&E  Organization  and  Customer  Interface. 


New  Program  (No  RTO  Exists) 


Existing  Program  (RTO  Designated  and  Functional) 

1.6.1.  If  you  plan  to  conduct  a  test  in  one  or  more  of  the  A-P-A  mission  areas  you  should  contact  the 
A-P-A  SFTC  office.  This  office  helped  document  the  A-P-A  T&E  process,  is  the  custodian  of  it,  and 
can  provide  assistance  with  its  implementation  for  your  program.  The  A-P-A  SFTC  office  also  has 
experienced  test  planners  to  help  you  define  cost-effective  testing  options  for  consideration  by  you 
and  your  test  customer.  Further,  it  has  investment  planners  familiar  with  current  and  future  test  facil¬ 
ities  and  capabilities  of  the  Air  Force.  You  should  take  your  customer  test  requirements  and  test  capa- 
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bility  needs  to  the  A-P-A  SFTC  office.  They  will  help  you  with  your  initial  test  planning  and  any 
needed  test  investment  guidance. 

For  assistance,  call  or  write  the  A-P-A  SFTC  office  as  listed  below: 

A-P-A  SFTC  Office 

(805)  275-9250  or  DSN  525-9250 
FAX:(805)  275-7135  or  DSN  525-7135 
AFFTC/XPS 
195  E.  Popson  Ave. 

Edwards  AFB,  CA.  93524-6842 

Other  SFTC  Offices: 

Armament/Munitions  AFDT/DRC 

C4I  101  W.  D  Ave,  Suite  125 

EW  Eglin  AFB  FL  32542-5495 

DSN  872-9650 
904-882-9650 

Space:  SMC/CUC 

160  Skynet  St.,  Suite  1536A 
Los  Angeles  AFB  CA  90245-4683 
DSN  833-2504 

1.6.2.  Working  with  the  A-P-A  SFTC  office  should  save  you  considerable  time  and  effort  and  help 
you  get  your  early  test  planning  started  in  the  right  direction.  Once  a  RTO  and  OTA  has  been  desig¬ 
nated  for  your  program,  the  RTO/OTA  will  assist  you  in  your  test  planning  and  test  support  needs  [per 
AFI 99-101,  Development  Test  and  Evaluation  [formerly  Air  Force  Regulation  (AFR)  80-14],  Decem¬ 
ber  1993  and  AFI  99-102,  Operational  Test  and  Evaluation,  (formerly  AFR  55-43)].  If  your  A-P-A 

test  program  is  a  new  start  or  modification/Preplanned  Product  Improvement  (P^I)  program  and  has  a 
Program  Management  Directive  (PMD),  per  AFI  99-101  and  AFI  99-102,  you  must  contact  the 
A-P-A  SFTC  office  and  ask  for  their  assistance  on  early  test  planning.  If  you  will  be  writing  or  revis¬ 
ing  an  A-P-A  related  Test  and  Evaluation  Master  Plan  (TEMP)  and  an  RTO  has  not  been  designated, 
you  must  also  contact  the  A-P-A  SFTC  office.  The  three  (3)  SFTC  offices  maintain  close  contact  with 
each  other,  therefore  if  you  are  not  sure  whether  other  SFTCs  should  be  involved,  contact  the  A-P-A 
SFTC  office  and  they  will  involve  other  SFTCs  if  needed. 
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Chapter  2 

A-P-A  T&E  PROCESS 

Section  B— Description  of  the  A-P-A  T&E  Process 

2.1.  General.  Test  and  Evaluation  is  conducted  throughout  the  acquisition  process.  Figure  2.1.  illus¬ 
trates  at  a  top-level  where  and  how  T&E  related  subjects  and  phases  relate  to  the  five  acquisition  phases 

and  milestones.  It  should  be  noted  that  P^I  mods/upgrades  typically  conducted  during  the  Production  & 
Deployment/Operations  &  Support  phase  may  include  any  or  all  elements  of  the  system  maturity  from 
concept  to  total  system  integration  tests. 

2.2.  Weapon  System  Development  Flow.  The  systems,  subsystems,  and  components  associated  with 
A-P-A  mission  areas  are  developed  through  a  combination  of  both  independent  and  interrelated  design 
and  test  activities.  During  development,  various  subsystems,  components,  and  functional  areas  must  be 
integrated  to  function  as  a  cohesive  unit  ~  the  weapon  system.  The  airborne  platform  associated  with  the 
weapon  system  is  the  air  vehicle  and  is  the  A-P-A  T&E  process’s  primary  focus.  Figure  2.2.  describes  the 
general  flow  and  integration  associated  with  a  typical  A-P-A  system  development.  The  boxes  represent  a 
notional  flow  and  can  extend  over  time.  A  secondary  focus  of  the  A-P-A  T&E  process  is  to  also  provide 
the  data  required  in  the  development  of  ground  mission  simulators,  weapons  system  trainers,  etc.  An 
additional  focus  is  on  the  testing  of  support  equipment  being  developed  for  the  weapon  system  already  in 
inventory  as  well  as  performing  compatibility  testing  on  this  equipment  with  new  systems.  This  testing 
will  require  a  dedicated  effort  and  should  be  part  of  Logistics  Test  and  must  occur  in  concert  with  the 
Prime  Mission  Equipment  tests. 

2.2.1.  During  aircraft  development,  the  various  functional  and  specialty  disciplines  which  comprise 
A-P-A  systems  must  work  closely  together  to  ensure  adequate  integration.  The  extent  of  integration 
largely  depends  upon  the  aircraft’s  system  architecture;  federated,  integrated,  or  integrated  suite, 
which  are  described  in  the  following  paragraphs  and  shown  in  Figure  2.3. 

2.2.I.I.  Federated  System.  The  various  subsystems  (radar,  nav,  flight  controls,  etc.)  function 
autonomously  and  only  interface  where  data  transfer  is  required  (for  example,  the  air  data  sub¬ 
system  provides  only  pressure  altitude  to  the  navigation  computer  for  baro  damping).  In  addition, 
each  subsystem  interfaces  (via  displays,  scopes,  etc.)  with  the  aircrew  separately.  The  F-4,  A-10, 
and  KC-135  are  examples  of  a  federated  architecture. 
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Figure  2.1.  T&E  Relationship  to  the  Acquisition  Process, 


11 


T  ■ 


tt 


Dcnufifirai&iL 

Uijl 

VAldatiim 


Eriivneoiii^  & 
Matnufutmin^ 
Bevdomflti 


I’ltaH  in  \ 

JfflASt  IV 

ItoduLdunN 

0peraiiixn5 

and  \ 

and 

D^cymerd  \ 

Siqypnt 

l>eiFiekp  jnent  T  uid  £  valuaiian  (pT  AE) 


GberaiiDiial  Test  and  £vahia.dimfOT&£l) 


Syrtefn  iL«quironenu 
SAviewOS 


A 

SRKl 


Rlalor 


A 

SKR2  y\^^g-|rtwn  Da ign  Rwiwv^S DE^ 


SDR 


RevHvs 

& 

Audios 


A 

.SRR3  , 

I  Amlin  h4iTD«*et1  Si^ww(P^ 


Fundian;alBudm«  C<r£iRUDraDtiDii 


Aftc)[4i«iBafdme  Ccvifeur^duix 


jA>^CritioftlDwpn  ReutewrCC^IR')  1 

CDR  I  I 

/\ped  kwUfl  0Ta.£  EtsiJ  has  C«itrfio|ition 


Cqucqd  Deagn 


7 


/  rtiMca/Dparajlc  / 


z; 


D«ifin  BefticDfient 


7 


%ysiEjn. 

MaturRy 


/  CaamJiaiot  T«ty  / 


SuLjyrtqfu  Tcris 


7: 


z; 


SyrtaaTfeito 


I  /  TotaMw.  Tertf  7  I 

I  I  I 

I  /  rwi«t«i  /  • 

•  /  QTa  /  ' 


12 


AFMAN99-110  3  JULY  1995 


Figure  2.2.  A-P-A  Weapon  System  Development  T&E  Flow. 
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2.2.1.2.  Integrated  System.  A  more  complex  architecture  than  federated  because  data  are 
shared  between  subsystems  [usually  via  Military  (MIL)-STD  1553  data  buses].  Each  subsystem 
has  a  central  processor  which  controls  the  data  flow  within  the  subsystem  (intra  subsystem  data 
flow  via  subsystem  bus)  and  also  communicates  with  other  subsystem  data  processors  (inter  sub¬ 
system  data  flow  via  system  bus).  Situational  awareness  is  enhanced  because  data  from  multiple 
subsystems  can  be  presented  to  the  aircrew  on  one  display.  Systems  utilizing  integrated  system 
architectures  tend  to  have  graceful  degradation  following  a  system  failure.  The  F-15E,  C-17,  and 
B-2  are  examples  of  integrated  systems. 

2.2.1.3.  Integrated  Suite.  Integrated  suites  are  charac-terized  by  common  executive  control  and 
shared  core  hardware  and  software  used  to  implement  all  required  functions.  What  was  referred 
to  as  a  system  in  the  federated  and  integrated  system  architectures  is  now  referred  to  as  a  function 
in  the  integrated  suite.  The  system  is  now  defined  as  the  total  hardware  and  software  resources 
used  to  implement  all  required  mission  functions.  Integrated  suites  often  use  common  data  pro¬ 
cessing  and  signal  processing  components  in  a  modular,  scalable  computer  architecture.  Prepro¬ 
grammed  and  collected  data  is  fused  in  the  central  processor  to  provide  air  vehicle  mission 
management,  mission  level  situational  awareness,  navigation,  targeting,  fire  control,  and  defen¬ 
sive  functions.  The  data  displayed  to  the  pilot  are  an  amalgamation  of  the  data  collected  and  pro¬ 
cessed  simultaneously  by  the  total  system  resources.  The  F-22  avionics  are  an  example  of  an 
integrated  suite. 

2.3.  System  Maturity.  The  following  is  a  system  maturity  discussion  which  highlights  the  major  events 

and  major  aspects  which  occur  during  successive  stages  of  an  air  vehicle’s  development  (also  refer  to  Fig- 
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ure  2.1.,  Figure  2.2.,  and  Figure  2.3.).  The  division  between  each  of  the  development  phases  described 
below  is  not  discrete.  Instead,  system  development  is  continuous  and  requires  strong  communication 
between  users,  designers,  testers,  etc.,  in  one  phase  to  the  designers,  tester,  etc.,  in  subsequent  phases  to 
ensure  smooth  development  transitions.  Exit  criteria  for  each  of  the  development  phases  is  provided  in 
the  Exit  Criteria  template  in  this  document.  References  in  Attachment  3  contain  information  regarding 
system  development  and  system  engineering. 

Figure  2.3.  System  Architecture  Types. 


2.3.1.  Concept  Phase.  During  the  system’s  conceptual  design  phase,  top  level  documents  [Opera¬ 
tional  Requirements  Document  (ORD),  MNS,  Cost  and  Operational  Effectiveness  Analysis  (COEA), 
Integrated  Logistics  Support  Plan  (ILSP),  etc.]  create  a  foundation  upon  which  general  A-P-A  system 
requirements  are  built.  Design  tradeoffs,  primarily  using  Modeling  and  Simulation  (M&S),  occur 
extensively  during  this  phase.  Measurement  Facilities  (MFs)  are  the  primary  tools  used  to  provide 
data  to  support  high  level  M&S.  See  Figure  A2.2.  in  Attachment  2  for  a  flowchart  of  the  concept/ 
design  refinement  process. 

2.3.2.  Component  Phase.  Once  the  initial  component  and  subsystem  requirements/designs  have 
been  determined  from  both  functional  and  allocated  baselines,  component  development  begins.  Dur¬ 
ing  this  phase,  design  refinement  is  most  prevalent.  The  basic  thrust  of  component  testing  is  to  ensure 
the  components  perform  within  subsystems  as  designed.  Representative  models  created  in  early 
phases  will  develop  along  with  the  system  itself  and  are  updated  using  test  results.  Besides  M&S, 
MFs  and  System  Integration  Laboratories  (SILs)  are  the  primary  T&E  resources  utilized  during  com¬ 
ponent  development.  See  Figure  A2.4.  in  Attachment  2  for  a  flowchart  of  the  component  develop¬ 
ment  process. 

2.3.3.  Subsystem  Phase.  As  a  system  progresses  through  the  development  phases,  assessing  the 
tested  system’s  performance  against  the  user’s  requirements  becomes  more  straightforward.  The  sub¬ 
system  phase  is  the  first  opportunity  to  test  at  a  level  near  the  user’s  requirements  without  excessive 
extrapolation  of  test  results.  Also  during  this  phase,  the  number  of  new  models  created  begins  to 
taper-off,  but  model  sophistication/complexity  continues  to  increase  as  well  as  the  test  sophistication/ 
complexity.  Generally,  for  a  new  system,  subsystem  level  testing  is  conducted  in  System  Integration 
Laboratories  (SELs)  and  Hardware-In-The  Loop  (HITL)  facilities.  See  Figure  A2.5.  in  Attachment  2 
for  a  flowchart  of  the  subsystem  development  process. 

2.3.4.  System  Development  Phase.  All  of  the  various  subsystems  and  components  associated  with 
the  three  A-P-A  disciplines/systems  merge  to  form  the  air  vehicle.  During  this  phase,  operational 
aspects  of  T&E  become  more  prevalent  than  during  previous  phases,  culminating  in  OT&E  certifica¬ 
tion.  For  a  new  system,  the  DT&E  goals  are  to  first  ensure  the  air  vehicle  is  safe,  then  verify  that  the 
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air  vehicle  performs  basic  functions  as  designed,  establish  system  limits,  resolve  anomalies,  and 
finally  to  obtain  a  preliminary  assessment  of  the  system’s  operational  capabilities,  and  logistics  sup- 
portability.  During  this  phase,  many  of  the  ORD  requirements  can  be  tested  directly  by  DT&E  and 
OT&E  testers  and  test  results  may  also  be  used  to  update  high  level  models.  Primarily,  Installed  Sys¬ 
tem  Test  Facilities  (ISTFs)  and  Open  Air  Ranges  (OARs)  are  used  in  this  phase.  See  Figure  A2.7.  in 
Attachment  2  for  a  flowchart  of  the  system  level  development  process. 

2.3.5.  Integrated  System  Phase.  This  phase  of  a  system’s  development  is  concerned  with  testing  the 
weapon  system  in  an  environment  which  as  close  as  practical  replicates  the  environment  in  which  the 
system  is  intended  to  operate.  The  goal  of  this  phase  is  to  directly  test  and  deliver  an  effective  and 
suitable  weapon  system  with  no  operational  deficiencies.  To  accomplish  this,  realistic  suitability  and 
effectiveness  testing  must  be  performed  using  mission-level  tasks  including  maintenance  and  logistics 
supportability,  aerial  delivery,  bombing,  suppression  of  enemy  air  defenses  (SEAD),  etc.  See  Figure 
A2.9.  in  Attachment  2  for  a  flowchart  of  the  integrated  system  development  process. 

2.3.6.  Fielded  System.  Once  the  weapon  system  is  fielded,  P^I  modifications  which  are  changes  to  a 
system  still  under  production  and  upgrades  which  are  changes  to  a  system  no  longer  under  production, 

require  T&E.  These  P^I  mods/upgrades  could  range  from  major  to  minor,  and  be  performed  at  either 
a  logistics  center,  laboratory,  or  major  test  center.  Obviously,  the  extent  of  T&E  required  and  there¬ 
fore,  the  resources  required,  depends  upon  the  exact  nature  of  the  mod/upgrade.  Unlike  a  new  system 

development  where  generally,  T&E  starts  with  M&S  and  ends  with  OARs,  P^I  mod/upgrade  pro¬ 
grams  can  start  at  any  T&E  resource  category  level.  The  six-step  T&E  process,  however,  remains  the 

same.  Component  testing  of  a  P^I  mod/upgrade  could  be  conducted  on  a  flying  test  bed  in  an  OAR 
without  first  going  through  a  SIL,  HITL,  or  ISTF  if  the  test  team  deems  the  risk  minimal. 

2.3.7.  Level  of  Integration  Between  Mission  Areas.  An  important  aspect  of  system  maturity  is  the 
level  of  integration  between  the  three  A-P-A  mission  areas  as  well  as  the  subsystems/systems  associ¬ 
ated  with  each  A-P-A  mission  area  required  at  each  phase  of  a  system’s  maturity.  Figure  A2.4.  shows 
generally  the  level  of  integration  between  A-P-A  mission  areas  as  the  weapon  system  development 
progresses. 


Figure  2.4.  Notional  Level  of  Integration  Between  A-P-A  Mission  Areas. 
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2.4.  Test  Process.  The  test  process  by  itself  is  not  the  whole  story  for  successfully  planning  and  execut¬ 
ing  a  test.  Bringing  the  right  test  tools/resources  to  bear  at  the  right  phase  in  a  system’s  development  is 
also  critical.  The  expertise  of  the  test  resources  should  be  exploited  as  early  as  possible  to  aid  in  control¬ 
ling  programmatic  risk.  For  example,  test  centers  can  facilitate  early  manufacturer’s  testing  by  recom¬ 
mending  and/or  loaning  data  acquisition  equipment  or  by  processing  specialized  data.  Figure  A2.5. 
illustrates  how  the  test  process  relates  with  test  resources  and  system  development  maturity. 


Figure  2.5.  Test  Process  Expanded  to  Cover  Other  Aspects  of  T&E. 
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Figure  2.6.  A-P-A  T&E  Cube. 
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2.4.1.  A-P-A  T&E  Process  Multi-Dimensionality.  The  A-P-A  system  development  involves  many 
simultaneous  design  and  test  efforts  of  the  A-P-A  functional  categories/disciplines,  each  at  varying 
degrees  of  maturity,  merging  to  form  a  cohesive  integrated  weapon  system.  Various  levels  of 
resources  are  also  used  during  each  phase  of  a  system’s  maturity.  To  more  concisely  capture  the 
multi-dimensionality  of  the  A-P-A  T&E  process.  Figure  A2.5.  is  modified  to  form  the  T&E  cube  pre¬ 
sented  in  Figure  A2.6.  The  main  essence  of  the  cube  is  that  the  T&E  process  is  performed  repeatedly 
as  a  system  develops.  It  is  performed  simultaneously  on  hundreds  and  thousands  of  components,  each 
at  varying  degrees  of  maturity  and  each  relying  upon  various  T&E  resources/facilities.  As  the  compo¬ 
nents  merge  to  form  subsystems,  the  test  process  is  again  implemented  repeatedly  on  each  subsystem, 
using  various  T&E  resources,  and  so  on  through  the  weapons  system’s  entire  development. 

2.4.2.  Requirement  Traceability.  As  the  system  develops,  fewer  tests  are  performed,  but  they 
become  more  sophisticated  and  address  higher  level  requirements.  During  the  early  phases  of  a  sys¬ 
tem’s  development,  the  test  results  are  compared  to  low-level  requirements  which  are  derived  from 
higher  level  requirements.  As  the  weapon  system  evolves  and  matures,  tests  are  performed  which 
address  requirements  at  higher  and  higher  levels,  until  finally,  test  results  can  be  compared  directly  to 
the  Operational  Requirements  Document  (ORD)  and  Cost  and  Operational  Effectiveness  Analysis 
(COEA)  data.  It  should  be  understood  that  not  all  parameters  will  compare  on  a  one-for-one  basis  due 
to  differences  between  what  is  placed  in  the  ORDs  and  what  is  placed  in  the  contract.  For  example, 
the  user’s  reliability  requirements  are  stated  in  operational  parameters  in  the  ORD  whereby  the  SPO 
will  express  their  requirements  in  contractual  parameters  with  proportional  (but  different)  quantitative 
values. 
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2.4.3.  Generic  Air  Force  T&E  Process.  The  generic  Air  Force  T&E  process  contained  in  AFI 
99-103  is  shown  in  Figure  A2.7.  To  implement  the  Air  Force  T&E  process  for  A-P-A  mission  areas, 
minor  modifications  were  made  to  it  which  further  clarify  the  A-P-A  mission  area  T&E  process  steps. 
The*  A-P-A  T&E  process  is  shown  in  Figure  A2.8. 

Figure  2.7.  Air  Force  T&E  Process. 
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Figure  2.8.  A-P-A  T&E  Process. 
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Figure  2.9.  Determine  Test  Objectives. 
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2.4.4.  A-P-A  T&E  Process  Steps.  The  following  section  is  a  brief  discussion  of  each  A-P-A  T&E 
process  step.  Additional  details  are  presented  in  the  Test  Process  template  in  section  3.2. 

2.4.4.I.  Determine  Test  Objectives.  Clearly  defining  the  test’s  objectives  will  bound  the  test. 
Use  program  documents  such  as  the  ORD,  TEMP,  Requirements  Correlation  Matrix  (RCM),  Inte¬ 
grated  Logistics  Support  Plan  (ILSP),  etc.,  to  define  requirements  and  help  establish  the  test  objec¬ 
tives.  As  part  of  developing  the  test  objectives,  ensure  the  test’s  success  criteria  are  defined  to 
know  when  testing  is  complete.  System  requirements  flow  top-down.  General  requirements, 
beginning  with  the  MNS,  ORD,  and  COEA,  are  broken  down  into  more  detail  in  system  specifica¬ 
tions  (type  A  and  type  B  specifications)  and  ILSPs.  The  specifications  in  turn,  then  drive  the  indi¬ 
vidual  aircraft  systems/subsystems  requirements. 

2.4.4. 1.1.  Detailed  Test  Planning  is  normally  done  by  the  RTO.  The  system  specification 
requirements,  operational  requirements,  basic  objectives,  and  pre-test  analysis  are  broken 
down  into  manageable  elements  that  allow  systematic  control  of  test  conditions  and  system 
(test  article)  configuration.  Test  methods  and  procedures  are  developed  to  limit  the  number  of 
variables  that  might  influence  the  result  of  a  test,  thereby  ensuring  repeatability  and  confi¬ 
dence  in  the  data.  The  analysis  plan  is  integrated  with  the  detailed  objectives  in  determining 
statistical  requirements  for  establishing  system  accuracy  and  other  performance  measure¬ 
ments.  The  detailed  objectives,  test  conditions,  test  article  configurations,  and  data  require¬ 
ments  meld  together  to  yield  a  matrix  of  test  points  or  discrete  test  events.  Detailed  test 
planning  must  also  include  a  strategy  for  reducing  performance  risk  as  quickly  as  possible 
while  considering  system  maturity  and  readiness  for  any  given  test.  Design  engineers  often 
know  of  potential  system  weaknesses  resulting  from  tradeoffs  made  early  in  engineering,  man¬ 
ufacturing,  and  development  (EMD).  Design  margins  are  typically  reduced  during  the  trade 
process  and  if  they  influence  critical  performance  requirements,  they  should  be  tested  as  soon 
as  possible.  Other  targets  for  early  risk  reduction  testing  include  new  technology,  software, 
system  interfaces,  and  component  reliability.  This  part  of  the  test  strategy  is  geared  toward 
developing  and  maturing  the  system  and  it  overlaps  the  second  part  of  the  strategy,  which 
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should  focus  on  testing  as  efficiently  as  possible.  Experience  is  a  key  factor  in  laying  out  an 
efficient,  detailed  test  event  schedule. 


Figure  2.10.  Conduct  Pre-Test  Analysis. 
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2.4.4.2.  Conduct  Pre-Test  Analysis.  Use  modeling  and  simulation  (M&S),  theoretical  analysis, 
previous  test  results,  engineering  judgment,  etc.,  to  predict  the  system’s  performance.  These  pre¬ 
dictions  should  be  developed  at  every  stage  of  system  maturity.  Developing  the  detailed  test  plan 
overlaps  pre-test  analysis  and  the  development  of  test  objectives  in  the  first  step.  Based  on  the 
data  requirements  in  the  detailed  test  plan,  a  configuration  of  test  instrumentation  should  be  iden¬ 
tified.  Evaluation  criteria  should  be  developed  with  a  basis  in  the  requirements  documentation. 
Specific  test  resources  should  also  be  identified.  The  matrix  of  test  points  should  be  refined  using 
statistical  tools  and  engineering  judgment  to  minimize  the  number  of  test  points,  resources,  and 
time  required  to  answer  the  objectives.  This  is  the  point  where  the  scope  of  the  test  and  the  overall 
test  schedule  are  completely  defined.  Until  this  point,  test  managers  can  use  schedule  models  with 
historical  data  from  similar  programs  to  develop  a  preliminary  schedule,  but  a  realistic  schedule 
depends  on  the  actual  work  plan  which  should  be  built  from  the  bottom  up.  The  schedule  must 
also  be  adjusted  for  expected  and  unexpected  delays  due  to  late  delivery  of  the  system,  system 
failure,  non  availability  of  support  resources,  redesign,  regression  testing,  unusable  data,  poor 
weather,  new  safety  hazards,  etc.  A  schedule  that  fails  to  consider  these  delays  is  unrealistic  right 
from  the  outset.  The  detailed  test  plan  is  formally  reviewed  by  a  group  of  experts,  normally 
referred  to  as  a  Technical  Review  Board  and  composed  of  senior  engineers  external  to  the  project. 
For  flight  tests,  a  similar  group  of  experts  reviews  the  safety  analysis  to  be  certain  that  hazards  are 
minimized  to  the  lowest  practical  level.  This  group  is  normally  referred  to  as  a  Safety  Review 
Board.  The  completion  of  both  boards  is  required  before  the  start  of  a  flight  test  and  may  be 
required  to  reconvene  if  new  tests  are  added  or  new  hazards  are  identified. 
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Figure  2.11.  Conduct  Test. 
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2.4.4.3.  Conduct  Test.  This  step  in  the  process  may  be  more  complex  than  it  appears,  depending 
on  the  phase  of  development.  It  is  usually  the  most  manpower-intensive  step  because  of  the  num¬ 
ber  of  activities  and  associated  workload  that  takes  place  in  the  short  time  period  prior  to  and  dur¬ 
ing  the  test.  Before  a  test  can  begin,  the  team  must  select  test  events  from  the  overall  matrix  of  test 
points  using  a  strategy  that  considers  system  maturity,  risk  in  system  design,  and  readiness  to  con¬ 
duct  specific  tests  at  specific  conditions.  As  a  minimum,  the  team  must  decide  on  the  types  of  test¬ 
ing  so  that  support  activities  can  proceed.  These  activities  include  the  scheduling  of  resources, 
preparation  of  the  test  articles,  finalizing  support  plans,  calibrating  instrumentation,  facility  set-up 
and  team  briefings.  Examples  which  occur  in  open  air  testing  are  control  room  set-up  and  opera¬ 
tion,  which  are  normally  supported  by  plans  or  instructions  designed  for  the  specific  tests.  These 
activities  are  so  integral  to  conducting  the  test  that  they  should  be  considered  part  of  the  test  exe¬ 
cution  itself.  In  some  cases,  these  preparatory  actions  are  test  events  in  their  own  right.  For  exam¬ 
ple,  pre-flight  of  a  test  aircraft  or  weapon  is  required  before  flight,  but  the  pre-flight  can  also  be 
part  of  the  system  suitability  evaluation.  The  selected  test  events  are  organized  according  to  vari¬ 
ous  planning  considerations  such  as  prerequisite  actions,  test  procedures,  system  loading,  build-up 
in  conditions  and  efficiency.  A  timeline  is  normally  developed  based  on  the  estimated  duration  of 
each  test  event  and  set-up  for  each  event.  This  execution  plan  can  be  written  in  various  forms 
depending  on  the  phase  of  testing.  For  example,  in  a  System  Integration  Laboratory  (SIL)  test,  the 
actual  test  and  system  operating  procedures  can  be  specified  in  detail,  one  after  the  other,  with  the 
test  conditions  and  other  requirements  for  set-up  integrated  right  in  with  the  procedures.  In  a 
flight  test,  the  team  normally  formulates  a  set  of  test  cards  that  provide  similar  information  while 
highlighting  limitations  and  potential  hazards.  The  execution  plan  is  a  critical  element  in  the  over¬ 
all  test  process  because  it  brings  everything  from  prior  planning  together  just  before  the  resources 
are  used.  It  establishes  disciplines  and  control  during  test  execution,  and  helps  to  ensure  safety, 
data  repeatability,  and  error-fi:ee  testing.  The  conduct  of  a  test  is  normally  governed  by  estab¬ 
lished  procedures,  practices  and  techniques  that  are  recognized  by  technical  experts  as  valid  test 
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methodology.  Occasionally,  new  test  methods  are  required  to  keep  pace  with  advancing  technol¬ 
ogy.  When  this  occurs,  the  test  and  engineering  communities  must  join  to  validate  the  methodol¬ 
ogy  prior  to  the  test.  Whatever  methods  are  used,  the  testers  must  be  trained  or  skilled  in  the 
procedures,  practices,  and  techniques  used.  Data  can  be  collected  in  a  variety  of  ways,  from 
human  observation  and  hand  recording,  to  programmable  systems  capable  of  megabit  recording. 
Testing  of  complex  systems  usually  requires  automated  instrumentation  with  speed,  resolution, 
and  recording  capacity  that  matches  the  system  under  test  (SUT).  When  the  performance  or  func- 
tionaUty  of  an  end-system  is  tested,  data  can  be  gathered  manually  but  tracing  the  cause  of  system 
anomalies  usually  requires  discrete  measurements  at  a  subsystem  level  and  precise  data. 

More  complicated  testing  occurs  as  the  system  development  progresses  from  early  M&S  to  labo¬ 
ratory  brassboard,  component,  subsystem,  and  full  system  testing.  Both  hardware  and  software 
undergo  testing.  During  the  early  phases  of  development,  they  occur  relatively  independently, 
then  merge  to  form  a  system  and  must  then  be  tested  as  a  unit. 


Figure  2.12.  Perform  Post-Test  Analysis. 


Determine 

Quality 


Process  Data  into 
Engineering  r 
F  onnats 


Perfonn  Post-Test 
Aitalj^sis 


Evaluate  System 


Performance  vs 
Threshold 
Ciiteiia  and 
Predictions 


Identify 
Deficiencies  in 
System  or  Test 


2.4.4.4.  Perform  Post-Test  Analysis.  This  step  consists  of  two  major  substeps:  post-test  data 
and  system  performance  evaluation.  The  data  must  be  converted  from  raw  form  to  a  usable  format 
in  engineering  units  before  it  can  be  analyzed.  Many  programs  have  found  it  beneficial  to  have  a 
"quick-look"  data  processing  capability  to  determine  data  quality  before  sending  it  forward  for 
more  elaborate  processing.  The  quick-look  capability  should  process  the  data  into  first-generation 
engineering  units  which  allow  an  early  view  of  system  performance  during  specific  test  points.  A 
Quick-Look  capability  is  very  useful  when  test  results  are  required  prior  to  proceeding  to  the  next 
event.  The  development  of  this  capability  requires  a  small  investment  early  in  the  program  and 
usually  pays  big  dividends  in  time  and  money  throughout  EMD.  Data  are  archived  according  to  a 
prearranged  filing  system  so  that  it  can  be  easily  recalled.  Requirements  for  preserving  data  are 
established  during  early  planning  based  on  the  needs  of  the  program.  Unusual  test  results  or  inci¬ 
dents  may  add  to  the  basic  library  requirements.  For  a  flight  test,  system  performance  data  may  be 
merged  with  time  space  positioning  information  (TSPI),  and  further  processed  using  analytical 
program  routines  to  yield  data  plots,  cross-plots,  and  tables.  The  system’s  performance  is  then 
evaluated  against  predicted  performance,  maturity  thresholds,  and  operational  requirements  crite¬ 
ria.  If  the  test  results  do  not  reasonably  match  the  predicted  performance,  a  check  should  be  made 
of  the  predictions,  test  methodology,  adherence  to  test  procedures,  data  quality,  and  analytical  rou- 
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tines  for  errors  or  factors  that  may  have  influenced  the  results,  such  as  weather  phenomena  or  elec¬ 
tronic  interference.  If  necessary  a  retest  is  conducted.  System  performance  and  reliability  are 
qualitatively  and  quantitatively  characterized  and  deficiencies  are  identified.  System  performance 
and  deficiencies  should  be  considered  with  perspective  toward  operation  requirements  and  tasks 
so  the  effect  of  a  performance  feature  of  deficiency  can  be  understood  in  terms  the  user  can  recog¬ 
nize. 

Figure  2.13.  Improve  Product. 


2.4.4.5.  Improve  Product.  If  the  results  of  the  evaluation  indicate  a  problem  exists  in  system 
performance  or  test  conduct,  action  is  required.  The  evaluation  should  provide  sufficient  informa¬ 
tion  to  determine  if  the  problem  resulted  from  an  error  in  the  test  process  or  the  SUT.  Once  under¬ 
stood,  problem  sources  within  the  test  procedures  should  be  eliminated  or  minimized  to  prevent 
future  occurrence.  This  may  require  changes  in  models/simulators,  test  methodology,  analytical 
routines,  etc.  The  system  can  be  retested  as  soon  as  the  test  process  is  improved. 

If  a  system  performance  deficiency  is  identified  in  the  evaluation  process,  the  path  will  probably 
be  more  difficult.  System  deficiencies  normally  require  design  improvements  and  these  improve¬ 
ments  take  time  to  implement.  For  example,  a  "simple"  software  change  can  cause  substantial 
redesign  and  regression  testing.  When  multiple  design  changes  are  made,  configuration  control 
becomes  critical.  Without  a  structured  and  disciplined  process  for  configuration  control,  safety  is 
compromised  and  cause  and  effect  relationships  become  impossible  to  determine.  The  path  for 
reporting  deficiencies  should  be  well  established  and  clear  of  obstructions.  Deficiency  reports 
deserve  everyone’s  full  attention  throughout  the  program.  The  strategy  for  introducing  improve¬ 
ments  in  the  design  depends  on  many  factors  which  could  differ  on  every  program.  The  best  tech¬ 
nical  strategy  may  not  be  in  line  with  the  best  business  strategy,  and  a  compromise  may  be 
necessary.  Only  the  program  manager  can  decide  the  best  course  overall.  When  design  changes 
are  introduced,  pre-test  analysis  should  be  condueted  before  conducting  more  tests. 
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Figure  2.14.  Report  Results. 
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2.4.4.6.  Report  Results.  Test  results  are  used  to  make  decisions.  During  the  system  develop¬ 
ment  phase,  they  help  design  engineers  decide  the  best  course  of  action  for  fixing  a  problem.  But 
the  biggest  decisions  on  a  program  are  the  milestone  decisions.  Early  in  a  program,  engineers  may 
have  to  choose  between  emerging  technologies  by  using  the  results  of  experiments  that  reduce  the 
risks  they  see  for  their  conceptual  system.  During  a  competitive  demonstration/validation  (Dem/ 
Val),  their  decision  will  determine  which  design  receives  go-ahead  for  EMD.  When  system 
development  has  progressed  to  the  point  that  the  system  is  operationally  representative,  a  decision 
is  required  on  its  readiness  to  undergo  full  operational  testing.  The  results  of  operational  testing 
provide  decision  makers  a  basis  for  their  production  decisions. 

Test  results  are  really  just  information,  which  means  test  results  can  be  communicated  in  countless 
ways.  Well  managed  programs  determine  the  type  and  frequency  of  reports  before  a  new  phase  of 
testing  begins.  It  is  not  unusual  for  results  to  be  communicated  daily  by  telephone  or  facsimile. 
Major  decisions  require  more  formal  reporting,  such  as  technical  letter  reports  or  preliminary 
reports  of  results.  The  latter  is  usually  given  in  the  form  of  a  briefing.  Final  technical  reports 
(TRs)  provide  historical  data  that  can  be  used  by  the  tester  for  future  tests  and  by  the  user  to  gain 
an  overall  insight  of  their  weapon  system.  The  substance  of  these  reports  varies  with  each  pro¬ 
gram,  but  normally  addresses  the  results  associated  with  each  test  objective  and  emphasizes  the 
operational  impact  for  the  user. 

2.5.  T&E  Resources.  This  portion  of  the  manual  is  a  discussion  of  T&E  resources  and  is  one  of  the 
three  sides  of  the  T&E  cube  on  Figure  A2.6.  Further  details  are  included  in  the  Resource  templates  pre¬ 
sented  in  section  3.2,  and  also  in  Attachment  2.  The  map  in  Figure  A2.9.  shows  the  location  of  major 
Air  Force  A-P-A  related  government  T&E  communities.  Note:  The  ALCs  are  primarily  dedicated  to 
operation  and  sustainment  (O&S)  testing  and  are  not  specific  development  test  assets.  The  facilities  cited 
are  sometimes  available  to  conduct  a  developmental  effort  dependent  upon  the  circumstances  and  level  of 
effort. 
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2.5.1.  The  various  T&E  facilities  perform  different  T&E  functions  such  as;  M&S,  laboratory  work, 
flight  test,  etc.  Following,  is  a  brief  description  of  the  six  DOD  recognized  T&E  resource  categories. 
Further  information  is  contained  in  the  Resource  templates,  (paragraphs  3.10.,  3.11.,  3.12.,  Attach¬ 
ment  3),  and  the  Air  Force  Test  Capability  Master  Plan  (TCMP)  and  Test  Investment  Strategic  Plan 
(TISP). 

2.5.I.I.  Modeling  and  Simulation  (M&S).  These  resources  are  used  in  every  stage  of  a  system’s 
develop-ment  to  help  design,  predict,  and  analyze  tests.  Generally,  during  the  concept  phase, 
M&S  form  the  basis  for  design  tradeoff  studies  and  are  at  a  high  level  i.e.,  threat  scenarios,  and 
environments.  As  the  system  develops  from  components  to  the  fielded  system,  M&S  can  be  used 
to  provide  generic  models  for  components,  interfaces,  etc.,  for  use  in  system  design  and  test. 
M&S  can  be  very  valuable  in  identifying  critical  system  areas  which  should  be  tested,  thus  reduc¬ 
ing  the  test  matrix.  Modeling  and  simulation  can  be  used  in  real-time  during  flight  test  as  a  system 
performance  comparison  tool  or  used  before  flights  as  a  platform  to  rehearse  upcoming  flights.  In 
either  application,  it  is  incumbent  upon  testers  to  ensure  model  assumptions  and  limitations  are 
clearly  understood  by  decision  makers.  M&S  results  can  easily  be  abused  if  the  underlying 
assumptions  and  limitations  are  ignored  or  unknown.  M&S  develops  with  the  system  by  using 
test  results  for  refinement  updates  and  by  progressing  from  low  level/extremely  detailed,  to  more 
high  level,  culminating  in  the  ORD/COEA  level  once  again.  Increased  reliance  upon  M&S  is 
encouraged  during  DT&E  and  OT&E.  M&S  examples  include;  real-time  pilot-in-the-loop  and 
in-flight  simulation  capabilities. 
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2.5.1.2.  Measurement  Facilities  (MF).  These  resources  provide  capabilities  to  measure  parame¬ 
ters  critical  for  system  design.  Generally,  they  are  not  test  article  unique,  but  do  perform  a  specific 
function  such  as  signature  and  navigation  accuracy  measurements  or  measurements  of  aerody¬ 
namic  forces.  A-P-A  Examples  include:  wind  tunnels,  signature  MFs,  spin-tunnels,  and  engine 
test  cells. 

2.5.1.3.  System  Integration  Labs  (SIL).  These  re-sources  integrate  hardware  and  software  com¬ 
ponents  up  to  the  subsystem  level  using  a  table-top/spread-bench  environment.  Usually  a  specific 
component  is  tested  in  a  SIL  with  the  interface  components  simulated  through  either  hardware  or 
software.  These  facilities  reside  at  contractor  facilities  because  they  are  test  article  specific  and 
are  required  for  integration  development.  They  exist  at  Air  Force  test  sites  for  integration  testing. 

2.5.1.4.  Hardware-In-The-Loop  (HITL)  Facilities.  These  resources  provide  the  capability  to 
test  actual  subsystem  level  hardware  in  a  closed-loop  indoor  lab  environment  such  as  iron-bird 
labs  and  fuel  system  labs.  Generally,  they  are  more  sophisticated  than  SILs  and  typically  reside  in 
contractor  facilities.  Subsystems  and  sensors  other  than  the  one  under  test  are  simulated.  A  gov¬ 
ernment  example  is  the  Integration  Facility  for  Avionic  Systems  Testing  (IFAST)  located  at  the 
Air  Force  Flight  Test  Center  (AFFTC),  Edwards  Air  Force  Base  (AFB)  CA. 

2.5.1.5.  Installed  System  Test  Facilities  (ISTF).  These  resources  provide  the  capability  to  eval¬ 
uate  systems  which  are  installed  on  the  host  platform/airframe  while  residing  within  a  controlled 
indoor  environment.  Examples  include:  anechoic  chambers,  limit  cycle/ground  resonance  test 
stands,  structural  loads  stands  and  full-scale  wind  tunnel  tests. 

2.5.1.6.  Open  Air  Ranges  (OAR).  This  category  of  T&E  resources  is  divided  into  two  groups, 
controlled/instrumented  ranges  and  real  world  environments.  Although,  the  weapon  system 
should  be  tested  in  a  controlled  environment,  valuable  data  may  be  gathered  from  real  world  sce¬ 
narios  which  may  effect  system  upgrades/modifications. 

2.5.2.  Figure  2.1.  shows  the  T&E  resource  type  generally  utilized  for  each  phase  of  new  system 
development.  T&E  resource  details  are  contained  in  the  A-P-A  templates  (section  3.2)  and  T&E 

Resources  (Attachment  3).  It  is  important  to  point  out  that  the  pace  of  development  of  a  P^I  program 
which  is  already  a  mature  and  fielded  weapon  system  is  very  dependent  upon  the  exact  modification/ 
upgrade  to  be  performed.  Considerations  for  program  risk,  safety  risk,  program  urgency,  etc.,  should 

also  be  factored  in  during  the  weapon  system  P^I  development  process.  Typically,  a  P^I  program 
relies  upon  an  abbreviated  number  of  resources  as  compared  to  a  new  program  acquisition. 


26 


AFMAN99-110  3  JULY  1995 


28 


AFMAN99-110  3  JULY  1995 


test  planning  by  ensuring  that  there  is  a  direct  link  between  MNS,  ORD,  TEMP  and  specification 
requirements  to  clearly  stated  test  objectives  and  quantifiable  measures  of  performance.  The  joint 
govemment/contractor  team  is  responsible  for  the  requirements  derivation  and  allocation  process  to 
ensure  traceability  and  acceptance.  The  SFTC  offices  described  previously  may  assist  in  correcting 
deficient  requirements  prior  to  RTO  designation  through  early  involvement. 

2.6.2.  T&E  Delaying  System  Development.  This  is  probably  one  of  the  most  commonly  voiced 
complaints  during  testing  and  usually  involves  many  facets  which  are  intertwined  and  in  many  cases, 
outside  the  tester’s  control.  Obviously,  the  user  would  like  the  needed  system  delivered  as  soon  as 
possible  and  meet  the  stated  requirements  documented  in  the  ORD.  The  developer/contractor  and 
SPO  do  their  best  to  meet  these  goals.  The  tester  bridges  the  sometimes  wide  gap  between  the  design 
engineers,  the  developer,  and  the  users.  Non-test  related  pressures  including  funding,  congressional, 
or  poorly  defined  requirements  and  test  related  pressures  such  as  failed  tests,  design  changes,  regres¬ 
sion  testing,  over-testing,  etc.,  force  schedule  slips  during  T&E.  The  schedule  may  have  been  unreal¬ 
istic  to  begin  with  due  to  inadequate  T&E  input  during  early  program  planning,  excessive  reliance 
upon  a  technology  which  did  not  materialize  or  test  schedules  which  did  not  account  for  various  test 
efficiency  factors.  In  any  event,  test  schedule  compression  usually  conflicts  with  one  of  the  tester’s 
main  charters  -to  conduct  an  adequate  system  evaluation  and  provide  decision  makers  with  factual, 
defensible,  and  repeatable  test  results.  Another  facet  to  the  tester  induced  delay  complaint  pertains  to 
the  real  world  pressure  to  fly-fly-fly  and  demonstrate  high  sortie  rates  without  adequate  regard  to  use¬ 
ful/productive  testing.  This  also  conflicts  with  the  tester’s  "evaluate"  objective.  In  an  attempt  to  get 
the  aircraft  out  to  the  user  as  quickly  as  possible,  some  of  the  systems  may  have  deficiencies  that  were 
not  discovered  during  T&E.  Other  common  sources  of  test  delays  are  late  delivery  of  Government 
Furnished  Equipment  (GFE)  items  needed  to  support  the  test  and  lack  of  adequate  test  instrumentation 
that  precludes  fault  isolation  to  a  sufficiently  low  depot  level  of  assembly.  It  is  a  well  documented  fact 
that  fixing  system  problems  late  in  the  acquisition  cycle  is  typically  very  costly.  To  minimize  these 
problems,  there  must  be  early  tester  involvement  in  the  acquisition  program  and  structured  develop¬ 
ment  testing  throughout  the  acquisition  process.  Overall  program  scheduling  should  provide  a  more 
realistic  assessment  of  the  test  schedule  as  part  of  the  program  executability  analysis  and  during 
source  selection.  The  schedule  must  be  driven  by  technical  requirements  and  the  capabilities  of  the 
various  government  and  contractor  agencies  to  fulfill  their  respective  roles. 

2.6.3.  Use  of  Test  Results  from  Immature  Systems.  This  problem  surfaces  when  comparing  test 
results  obtained  from  an  immature  system  against  the  user’s  requirements  (which  assume  a  mature 
system)  and  too  much  concern  is  levied  against  the  system  when  its  performance  is  below  the  opera¬ 
tional  requirements.  The  problem  is  compounded  by  the  push  to  conduct  operational  testing  earlier  in 
aircraft  development,  before  the  aircraft  systems  have  attained  a  sufficient  level  of  maturity.  AFO- 
TEC  uses  Operational  Assessments  (OAs)  early  in  DT&E  as  described  in  AFI 99-102  to  assess  major 
impacts  to  operational  effectiveness  and  suitability  and  to  identify  major  programmatic  voids 
adversely  impacting  the  ability  to  meet  operational  requirements.  These  assessments  are  intended  to 
evaluate  potential  areas  of  concern  for  operational  effectiveness  and  suitability.  The  problem  is  also 
compounded  in  software  intensive  aircraft.  By  its  very  nature,  software  changes  (as  its  name  implies), 
and  therefore  software  intensive  systems  have  a  more  difficult  time  achieving  maturity  and  a  stable 
configuration.  The  configuration  must  be  tightly  controlled  and  closely  tracked  during  T&E  to  accu¬ 
rately  evaluate  system  performance.  Engineering  judgment  and  historical  data  should  be  used  by 
testers  during  DT&E  and  especially  during  early  operational  assessments  (EGAs)  by  AFOTEC  to 
determine  whether  test  results  below  requirement  thresholds  are  truly  a  problem.  This  may  be  accom- 


AFMAN99-110  3  JULY  1995 


29 


plished  by  plotting  the  problem  parameter  (such  as  reliability)  against  time  to  indicate  trend  and  matu¬ 
rity  rate.  If  the  parameter  is  subjective  (such  as  handling  qualities)  or  cannot  be  plotted,  then  the  tester 
must  use  judgment.  Ensure  competent  software  development  personnel  are  in  place  and  have  a  firm 
understanding  of  the  function  and  system  environment  for  which  they  are  designing  the  software.  The 
System  Maturity  Matrix  (SMM),  reference  Acquisition  Policy  92M-001,  System  Maturity  Matrix, 
document  may  be  used  to  help  assess  whether  the  system  is  maturing  at  the  desired  rate. 

2.6.4.  System  Specification  vs.  User  Requirements.  This  is  a  common  occurrence  during  DT&E 
and  OT&E  and  surfaces  when  system  specifications  (specs)  do  not  align  well  with  the  user’s  require¬ 
ments.  This  also  occurs  when  the  user’s  requirements  are  ambiguous  or  always  changing  (may  be  due 
to  threat  changes).  If  a  system  meets  a  specification,  but  does  not  meet  an  ORD  requirement,  the  gov¬ 
ernment  is  faced  with  a  contractual  dilemma  and  results  in  a  gap  between  user  requirements  and  the 
original  system  specifications.  Disagreements  between  testers,  SPOs,  and  contractors  frequently 
result  which  polarize  the  involved  organizations.  The  tester  should  submit  a  deficiency  report  (DR) 
which  documents  the  deficiency  and  assists  in  resolving  the  issue  through  appropriate  channels.  To 
alleviate  the  problem  with  poor  specifications  or  misalignments  with  requirements,  the  tester  (RTO, 
OTA)  and  user  should  be  given  the  opportunity  and  adequate  time  to  review  the  system’s  specifica¬ 
tions  early  in  the  acquisition  cycle  or  when  the  contract  is  modified. 

2.6.5.  Realistic  Test  Planning.  Inadequate  resource,  schedule,  or  fiscal  planning  that  does  not 
account  for  regression  testing,  realistic  test  efficiency  learning  curve,  maintenance  down  time,  and 
unexpected  tests  can  cause  significant  problems  during  system  acquisition.  Care  must  be  exercised  by 
the  SPO  and  test  planners  to  not  be  overly  optimistic  during  test  program  planning  and  also  ensure  that 
when  pressures  are  applied  to  the  program,  these  factors  are  not  eliminated.  Too  often  when  this 
occurs,  cost  or  schedule  breaches  occur,  followed  by  a  continual  series  of  "get  well"  exercises  which 
rarely  succeed.  Aircraft  are  enormously  complex  machines  and  when  a  configuration  is  changed  due 
to  a  hardware  or  software  update,  modification,  or  correction  of  a  significant  deficiency,  some  regres¬ 
sion  testing  may  be  required  to  ensure  it  is  still  safe  before  proceeding  with  the  originally  planned 
tests.  Models  and  simulations  and  other  ground  based  resources  which  have  been  kept  current  with 
test  results  can  assist  in  regression  test  planning  and  reduce  the  number  of  test  points.  The  number  of 
regression  test  points  required  depends  upon  factors  such  as:  level  of  onboard  systems  integration, 
safety  risk,  and  extent  of  system  change.  The  cause  and  effect  relationship  involved  with  software 
intensive  aircraft  is  complex  and  requires  significant  analysis,  both  intra  and  inter  subsystems.  For 
instance,  a  seemingly  innocent  change  made  to  the  fuel  subsystem  may  have  a  drastic  effect  on  other 
subsystems  (avionics,  flight  controls,  etc.).  Regression  testing  should  go  through  the  system  maturity 
phases  (component,  subsystem,  etc.),  apply  the  six-step  test  process,  and  utilize  appropriate  T&E 
resources.  The  selection  of  test  sequences  should  be  based  upon  the  types  of  qualification  testing 
planned  for  qualifying  the  system,  the  type  of  equipment  eomposing  the  system  and  other  (e.g.,  safety 
consideration)  to  keep  the  types  of  retesting  of  corrective  fixes  to  a  minimum. 

2.6.5. 1.  As  stated  earlier,  test  planning  should  account  for  a  real  world  test  efficiency  of  less  than 
100  pereent.  In  fact,  a  good  historically  derived  planning  factor  for  real  world  test  efficiency  on  a 
major  weapons  system  is  approximately  60  percent.  The  actual  efficiency  depends  upon  many 
faetors  some  of  which  include:  aircraft  type,  safety  risks,  system  complexity,  and  time.  Test  agen¬ 
cies  should  use  historical  data  to  estimate  a  realistic  test  efficiency  and  utilize  this  data  in  TEMPs 
and  other  test  documents. 
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2.6.6.  Inadequate  T&E  Interfaces.  The  lack  of  a  consistent  test  point  of  contact  (POC)  during  sys¬ 
tem  development  is  frustrating  to  the  contractor  and  SPO  because  they  must  interface  with  numerous 
test  organizations  to  plan  and  conduct  tests.  They  need  and  want  a  single  POC  that  can  resolve  test 
issues  without  having  to  go  to  numerous  test  agencies  to  gather  the  whole  story.  The  A-P-A  SFTC 
office  should  help  mitigate  this  type  of  issue  prior  to  RTO  and  OTA  designation.  Once  an  RTO  and 
OTA  has  been  designated,  the  RTOs  and  OTAs  program  test  manager  [or  combined  test  force  (CTF) 
director,  if  RTO  is  organized  as  such]  should  serve  as  this  focal  point.  Another  related  concern  per¬ 
tains  to  test  support  interference.  Every  effort  must  be  made  by  the  test  support  organizations  to  not 
impinge  upon  test  conduct.  Testing  should  be  event-driven  resulting  in  completion  of  a  given  test 
point,  test  event,  or  series  of  tests  and  not  delayed  due  to  a  range  scheduling  conflict  or  failure  of  some 
test  support  element.  Early  involvement  of  the  test  centers  will  reduce  the  delay  caused  by  range 
scheduling.  However,  since  test  support  elements  sometimes  fail,  schedule  flexibility  must  be  a  part 
of  the  test  plan.  This  problem  is  not  easily  resolved  due  to  increased  funding  and  program  priority 
competitiveness.  With  limited  DOD  funds,  money  allocated  towards  system  development  means  less 
money  is  allocated  towards  T&E  support  resources.  For  example,  funding  may  be  required  for  updat¬ 
ing  T&E  "reference  source"  measurement  systems  accuracy  to  keep  abreast  with  onboard  system 
technology  advances.  Early  test  resource  planning  is  critical  in  ensuring  funds  are  utilized  efficiently 
and  where  most  needed. 

2.6.7.  T&E  of  Off-The-Shelf  Items.  In  recent  years,  the  DOD  has  given  greater  consideration  to 
procuring  systems  that  are  available  off-the-shelf  (OTS).  These  systems  offer  the  advantage  of  imme¬ 
diate  availability  with  no  investment  required  for  development.  An  even  greater  advantage  is  that  new 
OTS  systems  more  closely  reflect  the  state-of-the-art  with  technology  than  systems  that  are  developed 
under  the  government’s  lengthy  acquisition  process.  But  with  these  advantages  come  some  potential 
pitfalls  which  may  not  be  evident  to  the  naked  eye.  Commercial  Off  The  Shelf  (COTS)  systems  may 
or  may  not  be  suitable  for  military  applications.  The  best  way  to  determine  how  suitable  they  are  is  to 
test  them  in  the  required  mission  environment  under  realistic  conditions  before  deciding  to  procure 
them  in  quantity.  Practically  all  COTS  systems  that  are  considered  for  use  on  aircraft  must  interface 
with  the  aircraft  in  some  sense.  As  a  minimum,  they  need  electrical  power  and  therefore  must  be 
robust  enough  to  handle  the  small  surges,  spikes  and  momentary  interruptions  that  are  typical  of  air¬ 
craft  electrical  systems.  The  system  may  be  sensitive  to  background  noise,  structural  vibration  or  tem¬ 
perature  variations  that  occur  in  aircraft.  The  actual  mission  requirements  may  overload  the  system’s 
throughput  or  service  capabilities,  or  it  may  not  exhibit  the  robustness  and  reliability  needed  in  com¬ 
bat  conditions.  If  the  system  is  integrated  electronically  with  other  aircraft  systems,  engineering 
development  work  will  probably  be  needed  and  the  integration  will  need  testing.  This  is  particularly 
true  when  existing  software  is  changed  to  host  a  new  system.  Human  factors  may  also  be  important. 
For  example,  some  commercial  communications  systems  have  been  installed  in  aircraft  without  ade¬ 
quate  illumination  for  night  operations.  It  is  easy  to  overlook  a  potential  problem  if  a  thorough  eval¬ 
uation  isn’t  done  up  front,  including  an  assessment  of  what  testing  should  be  done  with  the  system 
fully  integrated  with  the  aircraft. 

2.6.7. 1.  Non-development  aircraft  acquisition  can  pose  greater  problems.  These  aircraft  are  usu¬ 
ally  tested  by  the  manufacturer  and  certified  by  the  Federal  Aviation  Administration  (FAA)  or  for¬ 
eign  agency  for  airworthiness.  A  thorough  examination  of  the  test  data  resulting  from  these  tests 
almost  always  shows  less  than  adequate  evaluation  for  military  mission  purposes.  It  is  not  enough 
to  simply  look  at  the  aircraft  specifications  and  advertised  capabilities  without  evaluating  the 
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existing  test  data  and  then  testing  the  airplane  as  it  is  intended  to  be  operated  in  its  military  mission 
environment. 

2.6.8.  Government  and  Contractor  Teaming.  The  roles  government  and  contractor  personnel  per¬ 
form  during  T&E  are  important  to  the  success  or  failure  of  a  test  program.  Generally,  successful  gov¬ 
ernment  and  contractor  teaming,  especially  during  DT&E,  results  in  more  effective  and  suitable 
aircraft  being  delivered  to  the  user.  The  level  of  T&E  teaming  depends  upon  what  T&E  phase  the  pro¬ 
gram  is  in  and  the  associated  level  of  government  test  involvement.  T&E  teaming  occurs  across  var¬ 
ious  facets/tasks:  managing,  planning,  conducting,  analyzing,  and  reporting.  The  tester’s  specific 
duties  (and  teaming  implementation)  at  a  given  point  in  time  are  dictated  by  these  facets  and  test 
phases. 

2.6.8. 1.  In  the  early  phases  of  program  development  prior  to  Responsible  Test  Organization 
(RTO)  or  Operational  Test  Agency  (OTA)  designation,  government  test  involvement  is  primarily 
limited  to  the  SPO  and  in  some  cases,  the  SFTC  offices.  During  these  early  phases,  besides  help¬ 
ing  to  solidify  system  requirements  and  assist  with  test  planning,  government  testers  (primarily 
SPO  engineering  or  test)  should  familiarize  themselves  with  the  system’s  design.  This  occurs 
through  familiarization  of  system  design  documents  (see  Documents  template  in  section  3.2), 
attending  various  program  design  reviews  [preliminary  design  review  (PDR),  critical  design 
review  (CDR),  system  requirements  review  (SRR),  etc.],  or  if  feasible,  attending  actual  tests. 
Building  a  strong  T&E  foundation  upon  extensive  knowledge  of  the  SUT  is  an  important  factor  in 
determining  the  success  or  failure  of  a  test  program. 

2.6. 8.2.  Once  the  RTO  and  OTA  has  been  designated,  the  level  of  T&E  participation  will 
increase.  A  small  cadre  of  experienced  test  personnel  and  aircraft  maintainers  form  the  core  for 
the  initial  test  force  (obviously,  the  number  of  people  in  the  small  cadre  depends  upon  the  test  pro¬ 
gram’s  size,  requirements,  and  schedule).  This  small  cadre’s  work  experience  includes  the  A-P-A 
disciplines  as  well  as  logistics/maintenance.  The  government  testers  bring  to  the  table  a  wider 
experience  base  and  operational  flavor  with  which  to  assist  the  contractor  and  ensure  the  system 
meets  the  user’s  needs.  As  stated  above,  this  small  cadre  should  familiarize  themselves  with  the 
system  design,  establish  contacts  (in  person  if  feasible)  with  the  SPO  and  contractor  (design  engi¬ 
neers,  testers,  logistics,  etc.),  and  assist  in  test  planning.  In  line  with  integrated  product  develop¬ 
ment  (IPD),  the  government  testers  should  be  included  as  members  of  the  appropriate  integrated 
product  teams  (IPTs).  In  performance  of  these  duties,  they  should  visit  the  contractor,  observe 
"key"  tests,  and  assist  the  contractor  in  test  planning.  It  is  vital  that  participation  be  constructive 
and  helpful.  As  such,  early  in  system  development,  while  the  system  is  still  under  design  refine¬ 
ment,  government  testers  should  be  documenting  system  deficiencies  or  problems,  primarily 
through  watch  items  [as  outlined  in  Technical  Order  (TO)  00-35D-54]  and  communication  to  the 
SPO.  The  contractor  will  be  much  less  likely  to  be  up  front  with  the  government  about  problems 
in  subsequent  phases  if  they  feel  they  were  "stabbed  in  the  back"  during  these  early  phases  before 
the  design  is  solidified.  Once  the  system  is  developed  and  nears  (or  in)  engineering,  manufactur¬ 
ing,  and  development  (EMD),  system  deficiencies  must  be  documented  using  DRs  and  watch 
items  in  accordance  with  (LAW)  TO  00-35D-54.  Government  and  contractor  interface  and  team¬ 
ing  responsibilities  must  be  defined  and  approved  by  the  Air  Force  Contracting  Officer  prior  to 
work  initiation,  as  appropriate. 

2.6.8.3.  In  performance  of  test  planning  duties,  DT&E  and  OT&E  personnel  will  ensure  they  are 
minimizing  duplication  by  using  common  data  requirements,  T&E  resources,  analysis  tools. 
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instrumentation,  etc.,  as  much  as  possible/practical  (OT&E  data  analysis  and  reporting  must  be 
lAW  Title  10,  United  States  Code,  Section  2399  and  Air  Force  policy).  If  government  and  con¬ 
tractor  test  points  are  integrated,  more  efficient  and  less  flight  testing  results. 

2.6. 8. 4.  During  test  team  buildup,  as  the  aircraft  development  progresses  toward  first  flight,  the 
test  force  grows  and  government  T&E  involvement  increases.  The  government  testers  (including 
logistics)  should  work  closely  with  the  contractor  in  detailed  test  plan  preparation  [even  if  a  con¬ 
tract  data  requirements  list  (CDRL)  item  to  the  government]  and  become  more  involved  with  tests. 
Again,  the  Air  Force  Contracting  Officer  must  approve  or  concur  with  such  participation.  Test 
involvement  examples  include  observing/participating  in:  component  and  subsystem  tests,  "key" 
system  tests  ("key"  as  determined  by  tester,  such  as  pilot-in-the-loop  simulator  tests,  surrogate 
testing,  etc.),  and  pre-flight  tests.  As  with  the  earlier  development  phases,  the  tester  will  note  and 
document  system  concerns/problems  through  watch  items,  but  now  also  scrutinize  them  to  deter¬ 
mine  which  warrant  upgrade  to  DR  status.  Contractor  participation  in  teaming  is  accomplished  if 
the  design  engineers  familiarize  themselves  with  testing  (test  plans,  conduct,  and  reporting). 
Design  engineer  involvement  in  tests  helps  ensure  a  strong  feedback  between  tests  and  design, 
which  in  turn,  improves  the  system’s  design  and  performance.  In  some  cases  (for  example  on  a 
large  effort,  complex  system),  contractors  are  encouraged  to  include  design  engineers  in  the  test 
control  room  during  testing. 

2.6.8.5.  As  the  aircraft  nears  first  flight,  government  involvement  in  safety  planning  also  begins 
to  increase  due  to  increased  accountability/responsibility.  Since  flight  testing  usually  occurs  on 
government  ranges,  government  safety  rules  must  be  accounted  for.  The  government  has  a  wide 
historical  base  to  call  upon  for  planning  safe  tests  and  managing  risk.  The  difficult  safety  planning 
task  for  the  government  is  to  not  overly  constrain  or  hinder  the  contractor,  yet  ensure  adequate 
safety  precautions  and  procedures  are  in  place  prior  to  test. 

2.6.8.6.  The  primary  method  to  ensure  teaming  during  flight  test  at  the  AFFTC  is  through  the  use 
of  CTF.  Although  other  test  centers,  agencies,  etc.,  may  not  currently  rely  upon  CTFs  to  perform 
T&E,  per  joint  USAF/TE  and  SAF/AQ  policy  guidance,  CTFs  are  the  recommended  approach  for 
T&E  as  a  way  to  reduce  duplications  (in  data  and  resources)  and  costs.  CTFs  integrate  DT&E, 
OT&E,  and  contractor  T&E  requirements  and  resources  (logistics,  engineering,  support  services, 
etc.)  under  one  organization  and  the  entire  test  process  (from  determining  test  objectives  to  test 
reporting)  is  implemented  within  one  organization.  One  of  the  most  important  aspects  of  flight 
test  is  to  provide  timely  test  result  feedback  in  the  form  of  reports,  DRs,  and  briefings  to  decision 
makers.  If  the  test  team  delays  this  feedback,  T&E  integrity  is  reduced  or  eliminated.  Also,  inde¬ 
pendent  DT&E,  OT&E,  and  contractor  reporting  is  an  essential  element  of  CTFs.  The  CTF  con¬ 
cept  may  not  apply  as  well  to  small  test  programs.  In  these  cases  the  test  organization  may  tailor 
the  CTF  concept  or  develop  a  test  team  approach  to  best  meet  their  requirements. 
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Chapter  3 

A-P-A  T&E  PROCESS  APPLICATION 
SectionC— Application  of  the  A-P-A  T&E  Process 

3.1.  General.  At  each  stage  of  an  acquisition  program  there  is  eonsiderable  pressure  to  reduce  T&E 
costs  and  schedule.  Test  program  planning  requires  trading  cost,  schedule,  and  test  content  such  as  num¬ 
ber  of  test  articles,  number  and  type  of  T&E  resources  utilized,  instrumentation,  etc.,  against  program 
risks  (safety  and  decision  confidence)  within  an  understood  environment  including  security,  fiscal  atmo¬ 
sphere,  political  atmosphere,  etc.  In  short,  each  program  must  be  tailored  to  minimize  cost,  schedule,  test 
content,  and  risks.  For  example,  ISTFs  or  HITLs  should  be  used  instead  of  more  robust  and  more  expen¬ 
sive  OARs  to  gain  adequate  confidence  in  a  given  system  if  the  ISTF  or  HITL  fulfills  the  T&E  require¬ 
ment.  Also  see  2.6.1.,  2.6.8.,  and  the  templates  in  section  3.2  for  additional  discussions  of  proper  test 
planning. 

3.1.1.  Not  achieving  weapon  system’s  performance  requirements,  within  schedule  and  cost  con¬ 
straints  as  outlined  in  program  documents  and  test  plans  will  significantly  increase  the  potential  for 
Congressional  oversight  and  program  reductions/cancellation.  There  are  numerous  ways  to  monitor 
the  test  portion  of  the  program’s  cost,  schedule,  content,  and  risks.  For  example,  if  validated  models 
using  test  results  are  utilized  in  a  program  development,  fewer  T&E  resources  may  be  required  overall 
because  models  can  more  quickly  and  cheaply  explore  the  test  envelope.  Also,  as  discussed  previ¬ 
ously,  sophisticated  HITLs  and  ISTFs  may  be  used  in  place  of  more  costly  and  time  consuming 
OARs.  Although  this  approach  may  reduce  some  of  the  system  development  time,  caution  should  be 
exercised  not  to  reduce  the  test  schedule.  Short,  compressed  test  schedules  are  rarely  ever  a  panacea, 
unless  the  SUT  is  mature  and  well  understood.  Short  schedules  at  first  may  appear  to  save  money,  but 
usually  in  the  end  are  more  costly.  Extra  time  and  money  are  required  which  address  increased  safety 
and  decision  risks  to  mitigate  program  problems  or  worse,  program  cancellation.  Therefore,  testers 
should  be  given  adequate  time  early  in  a  program’s  acquisition  cycle  to  review  and  comment  upon 
ORDs  and  TEMPs.  They  should  review  them  for  adequacy  and  testability.  Correcting  an  unrealistic 
schedule,  requirement,  or  test  plan  early  will  save  a  significant  amount  of  money  and  effort  later. 
Early  identification  of  performance  problems  also  allows  fixes  to  be  incorporated  into  the  system 
before  the  design  is  mature.  Designing  and  applying  a  corrective  action  early  is  far  less  costly  in  terms 
of  both  time  and  money.  Figure  3.1.  shows  the  influence  of  program  phase  upon  life-cycle-cost  deci¬ 


sions. 
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Figure  3.1.  Life-Cycle-Cost  Decision  Impacts. . 


,1: 


3.1.2.  The  test  planner  should  develop  a  robust  test  strategy  that  includes  numerous  fall-back  options 
in  the  event  that  undesirable  or  unpredicted  test  results  occur  during  testing.  The  strategy  should  be 
designed  to  support  decision  maker’s  needs,  identify  and  correct  deficiencies  early  in  development, 
and  include  the  T&E  necessary  to  support  evaluation  of:  user’s  key  parameters,  critical  system  char¬ 
acteristics,  COEA  identified  sensitive  parameters,  and  parameters  considered  risky.  As  part  of  the  test 
strategy,  identify  its  critical  path  and  also  identify  options  which  are  dependent  upon  test  results.  The 
critical  path  links  tests  which  directly  support  fixed  milestone  decisions  (for  example,  data  which  sup¬ 
port  a  fixed  schedule  budget  decision),  links  programmatic  risky  tests  (schedule,  cost,  performance, 
etc.),  and  links  key  decision  point  tests  including  tests  which  support  acquisition  milestones,  tests 
which  support  choice  of  one  technology  implementation  versus  another,  key  parameter  demonstra¬ 
tions,  etc.  It  is  helpful  if  the  tests  identified  in  your  test  strategy  address  issues  such  as  conducting  a 
series  of  small  tests  designed  to  obtain  results  quickly  and  early,  versus  one  big  test  late  in  the  acqui¬ 
sition  with  a  limited  or  no  fall-back  position. 

3.1.3.  Properly  planning  a  test  program,  be  it  ground/logistics/flight,  to  optimize  the  tradeoffs 
between  cost,  schedule,  test  content,  and  risk  is  centered  around  correctly  implementing  the  test  pro¬ 
cess.  T&E  resource  requirements  should  be  smartly  tailored  during  each  phase  of  a  system’s  develop¬ 
ment.  Test  results  obtained  from  these  T&E  resources  should  be  used  as  building  blocks  for  future 
tests.  The  templates  presented  in  section  3.2,  as  well  as  other  sections  of  this  manual  should  provide 
assistance  in  this  effort.  Each  test  should  adequately  stress  the  component,  subsystem,  etc.,  against  its 
requirements  and  confidence  should  be  gained  that  the  system’s  performance  is  adequate  before  it 
progresses  to  the  next  development  step.  The  goal  of  T&E  is  finding  and  correcting  problems  early. 
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Keep  in  mind  that  generally,  maximizing  the  use  of  M&S  (digital  pilot-in-the-loop)  and  in-flight  sim¬ 
ulation  and  ground  testing  (in  SILs,  HITLs,  and  ISTFs)  prior  to  and  during  flight  tests  reduces  overall 
test  costs,  since  open  environment  flight  tests  are  the  most  costly  portion  per  capita  of  the  T&E 
resource  categories.  This  relationship  is  shown  in  Figure  3.2. 


Figure  3.2.  Relative  Cost  -  T&E  Resource  Utilization. 


Relative 

Cost 


T&E  Resource  Type 

3.1.4.  At  ASC,  an  Integrated  Risk  Management  Process  (IRMP)  is  being  developed  and  implemented 
for  placing  emphasis  on  early  definition  of  program  tasks  and  their  interdependencies  with  more  real¬ 
istic  risk  assessments,  identification  of  constraints,  achievable  schedules,  and  reasonable  cost  esti¬ 
mates  at  all  stages. 

3.1.5.  Several  key  elements  in  the  IRMP  include  the  Integrated  Master  Plan  (IMP),  Integrated  Master 
Schedule  (IMS),  System  Maturity  Matrix  (SMM),  and  Technical  Performance  Measurement  (TPM). 

3.1.6.  The  IMP,  that  is  developed  during  the  demonstration  validation  phase,  identifies  program  mile¬ 
stones  and  success  criteria  that  must  be  satisfied  to  complete  the  program  within  the  defined  risk 
boundaries.  The  IMS  expands  the  IMP  to  the  work  planning  level,  defining  the  tasks,  their  duration 
and  interdependencies  required  to  complete  the  program.  These  documents  should  be  expanded  for 
each  program  phase,  used  to  develop  the  contract  basehne  and  to  task  all  government  and  contractor 
tasks.  In  parallel  with  the  initial  IMP  a  SMM  is  developed  to  capture  the  technical  performance 
parameters  in  the  requirements  correlation  matrix.  Systems  Requirement  Document  (SRD)  and  speci¬ 
fication.  The  metrics  including  Technical  Performance  Measures  (TPMs),  used  to  measure  progress 
are  developed  in  concert  with  industry.  The  test  planner  is  an  integral  part  of  this  entire  process.  The 
TEMP  is  developed  and  interfaces  with  the  IMP/IMS.  Detailed  test  planning  is  involved  during  the 
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entire  process  at  each  phase  of  development.  Additional  details  on  integrating  T&E  with  Acquisition 
Risk  Management  can  be  found  in  Attachment  1 . 

3.1,7.  In  summary,  a  well  tailored  test  program  should  have  an  auditable  building  block  of  T&E 
results,  balance  robustness  against  cost  and  schedule,  strive  to  identify  deficiencies  as  early  as  possi¬ 
ble,  and  use  validated  M&S,  where  applicable,  in  place  of  other  more  expensive  T&E  resources  to 
identify  the  critical  high  risk  issues  and  also  help  reduce  test  matrices.  For  more  details  on  all  three 
aspects  (T&E  process,  resources,  and  system  maturity)  of  the  A-P-A  T&E  process,  see  3.2.  of  this 
manual.  Attachment  2  of  this  manual  presents  a  more  detailed  discussion  of  weapon  system  devel¬ 
opment  using  the  T&E  cube  to  present  the  multi-dimensional  aspects  of  the  T&E  process.  It  includes 
refinements  and  details  of  the  test  process  and  resources  applied  at  each  stage  of  a  system’s  develop¬ 
ment. 

3.2.  Mission  Area  Templates.  This  section  presents  details  of  the  Airframe-Propulsion-Avionics  test 
and  evaluation  (T&E)  process  implementation  through  application  of  a  work  breakdown  structure  (WBS) 
type  hierarchy,  combined  with  a  template  type  format  (similar  to  Willoughby  Templates  referenced  in 
Attachment  1).  Efforts  to  implement  the  "Willoughby  Templates"  approach  in  documents  such  as:  DOD 
4245.7-M,  NAVSO  P-6071,  Best  Practices  (Dept,  of  the  Navy  manual).  Mar  86,  and  most  recently.  Dep¬ 
uty  Assistant  Secretary  of  the  Air  Force  Memorandum,  Templates  for  Certification  of  Readiness  for  Ded¬ 
icated  Operational  Test  and  Evaluation  (OT&E),  September  9, 1993,  have  been  very  successful.  Based 
on  this  fact  and  because  the  template  format  is  user  friendly  (it  allows  the  reader  to  quickly  gather  neces¬ 
sary  information  pertaining  to  the  A-P-A  T&E  process  implementation  at  a  level  of  detail  more  accessible 
to  the  reader),  the  template  format  has  been  applied  to  this  manual.  The  template  format  also  uses  bullet- 
ized  statements  to  most  efficiently  emphasize  and  capture  the  important  points. 

Details  of  the  following  subjects  are  contained  within  the  following  templates: 

1.  A-P-A  T&E  Process 

•  Determine  test  objectives,  pre-test  analysis,  test,  post-test  analysis,  improve  product,  report  results 

2.  Mission  Requirements  and  Integration 

•  Logistics  suitability,  interoperability,  susceptibility/vulnerability,  live  fire  tests  (LFT),  climatic 
tests,  human  factors,  safety,  and  flight  envelope/flight  manual  development.  Aircraft  Stores  Certi¬ 
fied  Program 

3.  Documents 

•  Acquisition/requirement  documents  [Operational  Requirements  Document  (ORD),  Cost  and 
Operational  Effectiveness  Analysis  (COEA),  SMM,  ILSP] 

•  System  configuration/engineering  docu-ments  [specifications,  Interface  Control  Documents 
(ICDs),  configuration  documents] 

•  Test  documents  [Test  and  Evaluation  Master  Plan  (TEMP),  detailed  test  plans,  DR,  analysis 
reports] 

4.  Embedded  Software 

•  Guidance,  determine  test  objectives,  pre-test  analysis,  test,  post-test  analysis,  improve  product, 
report  results 

5.  Exit  Criteria  (for  each  development  phase) 


AFMAN99-110  3  JULY  1995 


37 


•  Concept,  component,  subsystem,  system,  integrated  system,  and  fielded  system 

6.  A-P-A  T&E  Resources  (links  tests  to  resources,  however  resource/facility  details  are  in  Attachment 

3) 

3.2.1.  Each  template  is  stand  alone.  The  templates  are  universal,  for  use  by  program  managers, 
testers,  deeision  makers,  etc.,  and  therefore,  are  not  easily  tailorable  to  specific  functions  and  person¬ 
nel.  One  reader  may  be  a  program  manager  and  want  information  regarding  test  strategy  formulation, 
in  which  case  he  or  she  may  need  to  review  the  majority  of  the  templates  to  collect  the  big  pieture. 
Another  reader  may  be  a  tester  in  a  laboratory  and  desire  specific  information  on  what  to  be  concerned 
with  in  reviewing  a  TEMP.  Although  the  templates  may  overlap  one  another  in  some  areas  [for  exam¬ 
ple,  the  Test  Process  template  (test  planning  section)  overlaps  with  the  Documents  template  in  the 
subject  of  documents  review],  they  do  not  flow  from  one  to  another  (i.e.,  a  given  template  is  not  a  sub¬ 
set  of  another).  Figure  3.3.  presents  an  overview  of  what  information  is  provided  within  each  tem¬ 
plate  and  their  general  format.  Figure  3.4.,  Figure  3.5.,  and  Figure  3.6.  merely  list  what  disciplines 
are  eontained  within  each  A-P-A  mission  area. 
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Figure  3.3.  A-P-A  Templates. 

Templates  Cemmcm  to  All  Three  A-P-A  Mission  Areas 
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Within  these  templates,  sngg^tions,  he^fulhijits,  guidance,  etc.,  is 
provided  during  each  of  the  tbllirwing  T&E  st^s; 

..  Guidance  (from  DOD,  AF,  local  test  agency,  etc.) 

"  Requirements  Review  (What  documents  to 
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provided  for  the  folb^ving  type 
documents: 

-  Reqm'ts/ Acquisition 

-  Sys  ConfigfEngineering 

-  TestRebted 

Within  tins  ten^bte  the  Embedded 

Software  test  process  is  outiinedwith 
suggestions  on  items  to  he  considered 
or  included  for  eachstep: 

-  Guidance 

»  Determine  Test  Objectives 

-  Conduc  t  Pre-Test  Ana^'sis 

-  ConductTest 

-  Perform  Post-Test  Analysis 

-  Inqt rove  Product 

-  Report  Results 

Within  this  tempbte,  criteria  are 
identified  to  exit  ficom  one 
dwebpmentphase  to  another, 
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Integrated  System 
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Templates  Unique  to  A-P-A  Individual  Mission  Area 
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Within  each  setoften^btes,  examples  are  gven  for  types  of  tests  conducted  and  resources/focikties  utOiied  for  each  T&£  Reliance  Category 
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Note:  Resource/Facility 
details,  POC,  etc.,  are 
provided  in  attachment  3. 
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Figure  3.4.  Airframe  Disciplines/Systems. 


3.3.  Airframe  Disciplines/Systems; 

3.3.1.  Flying  Qualities.  Static  and  dynamic  stability  (longimdinal  and  lateral-directional),  maneuver¬ 
ing  flight,  high  angle-of-attack  (AOA),  handhng  qualities  and  stability  derivative  identification,  and 
transients  between  flight  control  modes. 

3.3.2.  Flight  Controls.  Control  laws,  system  stability  (gain/phase  margins),  control  surface/fin  actu¬ 
ators,  air  data  interface,  avionics  interface,  hydraulics  interface,  ground  collision  avoidance  systems 
(GCAS),  and  flight  control  portion  of  automated  terrain  following/terrain  avoidance  (TF/TA)  and  fire 
control/weapon  delivery  steering. 

3.3.3.  Structures.  Static  and  dynamic  loads,  weight  and  balance,  strength,  life,  material  properties, 
aeroelasticity,  and  vibroacoustics 

3.3.4.  Performance.  Range,  endurance,  speed,  payload,  thrust/drag  measurement,  air  data  correc¬ 
tions,  and  static  aerodynamics 

3.3.5.  Mechanical  Systems.  Environmental  control  and  protection,  hydraulic,  electrical  power  gen¬ 
eration/distribution/regulation,  fuel,  pneumatic,  miscellaneous  doors  actuation,  wings/fins/engine 
inlet  deployment  systems,  and  cooling/heat  exchangers,  and  airframe  general  interference  (engine 
installation  provisions,  canopy  function  and  seals,  etc.) 

3.3.6.  Crew  Station.  Controls  and  displays,  crew  station  and  passenger  compartment  layout,  egress/ 
ejection  systems,  and  life  support  systems  interface 

3.3.7.  Armament.  Weapon  captive  carry,  weapon  separation,  gunnery,  SEEK  EAGLE  (SE)  certifica¬ 
tion,  armament  (mechanical/electrical/hydraulic)  systems 
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3.3.8.  Mission/Auxiliary  Systems.  Air  refueling,  aerial  delivery  (cargo),  parachute  recovery  sys¬ 
tems,  accessory  drive  gearboxes,  landing  gear,  arresting  systems  (brakes  and  deceleration  devices), 
and  lighting  systems 

Figure  3.5.  Propulsion  Disciplines/Systems. 


3.4.  Propulsion  Disciplines/Systems: 

3.4.1.  Powerplant.  Performance  (Thrust,  specific  fuel  consumption,  fuel  flow  determination,  etc.) 
Operability  (surge  margin,  power  transients,  installation  effects,  flight  envelope,  start  envelope,  com¬ 
ponent  interfaces,  etc.),  Durability  [Low  cycle  fatigue  (LCF),  High  cycle  fatigue  (HCF)],  ingestion 
(birds,  rain,  ice,  gun  gas,  rocket  exhaust,  steam,  foreign  objects),  and  maintainability. 

3.4.2.  Inlet.  Performance  (mass  flow,  pressure  recovery,  flow  distortion,  etc.).  Operability  (starting, 
unstarting,  geometry,  bleeds,  ingestion,  acoustics,  etc.).  Durability  (LCF,  HCF,  etc.),  and  Maintain¬ 
ability. 

3.4.3.  Nozzle.  Performance  (mass  flow,  discharge  coefficients,  etc.).  Operability  (vectoring,  control 
response,  observables,  acoustics,  etc.).  Durability  (LCF,  HCF,  etc.),  maintainability,  observability, 
nozzle/afterbody  drag,  thrust  vectoring/reversing  and  interfacing  with  airframe. 

3.4.4.  Auxiliary  Power.  Auxiliary  power  unit  (APU),  jet  fuel  starter  (JFS),  starting/operating/inter- 
face  characteristics,  and  secondary  power  systems. 
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3.5.  Avionics  Disciplines/Systems: 

3.5.1.  Sensors.  Radar,  electro-optical  (EO)  sensors/trackers/designators,  infrared  (IR)  sensors/  track¬ 
ers/  designators,  millimeter  wave  (MMW)  sensors,  lasers,  ultraviolet  (UV)  sensors,  multi-spectral 
sensors,  reconnaissance  sensors,  air  data  systems  (airspeed,  temp,  Mach)  and  ice  detection. 

3.5.2.  Communications,  Navigation,  and  Identification; 

3.5.2.1.  Communications/Identification.  Radios  ultra  high  frequency/very  high  frequency/high 
frequency  (UHF/VHF/HF)  and  data  links.  Satellite  Communication  (SATCOM)  systems.  Identifi¬ 
cation  systems,  beacons/transponders,  telemetry  systems,  range  safety  receivers/decoders. 

3.5.2.2.  Navigation.  Inertial  navigation  systems  (INS),  magnetometers,  Doppler  velocity  sys¬ 
tems,  radar,  radar  altimetry.  Global  Positioning  Systems  (GPS),  Tactical  Communications  and 
Navigation  (TACAN),  Long  Range  Navigation  (LORAN),  Instrument  Landing  Systems  (ILS) , 
Digital  Terrain  Systems,  night  vision  systems. 

3.5.2.3.  Guidance.  Signal/power  distribution  units,  mission  control  modules,  mission  manage- 
ment/Operational  Flight  Program  (OFP)  systems,  mission  computer. 

3.5.3.  EW  Integration.  Integration  with  avionic  systems  and  other  aircraft  subsystems  (for  example, 
flight  controls)  in  support  of  electronic  warfare  (EW)  mission  functions. 

3.5.4.  Weapon  Systems.  Weapon  delivery,  fire  control  systems,  smart  weapon/missile  integration 
with  avionics  systems,  stores  management  systems  (SMS)/interfaces,  air  vehicle/weapon  OFP  soft¬ 
ware. 

3.5.5.  Support/Mission  Systems.  Controls  and  displays,  mission  computers/control  modules,  elec¬ 
trical  power  distribution  systems,  avionics  self  test/built-in-test  (BIT),  onboard  simulation/training 
systems,  helmet  mounted  displays  (HMDs),  offboard  support  systems  usually  developed  integral  with 
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UAV/cruise  missile  system  operation  (ground  or  airborne  launch  systems,  mission  data  planning  sys¬ 
tems  and  ground  or  airborne  remote  flight  command/mission  control  systems). 


Figure  3.7.  Test  Process  Template 
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Test  Process 
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Resources 


Detenuine  Test  Objectives 
Conduct  Pte-Test  Analysis 
Conduct  Test 

Perform  Post-Test  Analysis 
Improve  Product 
Report  Results 


3.6.  Test  Process  Template:  See  Resource  templates  for  what  type  of  tests  are  performed  during  each 
development  phase  and  what  resources  are  utilized. 

3.6.1.  Determine  Test  Objectives: 

-Review  requirement  documents  (see  Docu-ments  template) 

—Concept  phase 

— Review  ORD,  COEA,  and  MNS 

— Identify  sensitivity  of  system  performance  to  key  parameters  (see  DOD  Directive 
5000.2-M,  and  API  10-601  for  definition) 

—Component  through  System  Development  Phases 

— Review  functional,  allocated,  and  product  specifications,  ORD,  and  COEA 

—Integrated  and  Fielded  System  Phases 

— DT&E  should  review  OT&E  test  plans  and  ORD  (to  ensure  realistic  operational  testing) 

— Involve  your  test  center  early  in  the  identification  of  test  purpose  and  test  plans. 

-Identify  the  purpose  of  test  and  formulate  the  test  plans  (see  DOD  5000  series  guidelines  and  Exit 
Criteria  template  for  test  planning  guidance) 

—Concept  Phase 

— Tests  primarily  support  concept  tradeoff  study  models/simulations  or  demonstrate  new 
technologies  being  considered  for  the  system’s  design 

—Component  through  System  Development  Phases 

— Generally,  tests  are  conducted  to  assess  an  item’s  functionality  against  its  design  (to 
ensure  airworthiness,  assess  its  performance  against  specification,  preliminary  evaluation  of 
operational  effectiveness  and  suitability,  etc.) 
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— Tests  become  more  integrated  between  disciplines  and  as  such,  interfaces  (software  and 
hardware)  must  be  evaluated 

-Integrated  and  Fielded  System  Phases 

— DT&E  should  expand  flight  and  onboard  systems  envelopes,  develop  TOs  (flight  and 
maintenance),  and  conduct  tests  to  gain  confidence  in  OT&E  certification  (assess  mission  level 
requirements) 

— OT&E  evaluates  operational  effectiveness  and  suitability 

-Identify  the  test  strategy’s  critical  path  (path  which  supports  evaluation  of  user’s  key  parame¬ 
ters,  critical  system  characteristics,  COEA  identified  sensitive  parameters,  and  high  program  risk 
areas) 

-Identify  test  success  criteria 

-Consider  decision  maker’s  and  user’s  needs 

-Who  is  going  to  use  the  data  and  what  decision  will  data  support? 

—Are  data  for  System  Program  Office  (SPO),  other  testers/designers.  Office  of  the  Secre¬ 
tary  of  Defense  (OSD),  etc.? 

—Are  data  for  final  report,  milestone  (MS)  approval,  system  engineering,  etc.? 
-Suggestion:  Draft  a  rough  outline  of  test  report  to  help  ensure  you  have  answered  the  mail 
3.6.2.  Conduct  Pre-Test  Analysis: 

-Confirm  that  the  configuration  of  the  SUT  is  consistent  with  test  objectives 
-Predict  System  Performance 

-Review/use  previous  test  results  and  similar  systems  data  to  help  predict  system  performance 
—Use  engineering  judgment  and  past  experience 

-Use  requirement  documents  [ORD,  ILSP,  Logistics  Support  Analysis  (LSA)  and/or  specifica¬ 
tions]  to  help  focus  the  test  predictions 

-Use  M&S  as  a  tool  to  predict  system  performance 

-Identify  test  conditions  and  prepare  test  plan 

-Rely  upon  statistical  analysis  tools  and  sound  engineering  judgment  to  minimize  test  matrix 

— Use  industry  accepted  method  (i.e.,  Tachuchi)  and  perform  a  “Design-of-Experiment”  which 
will  define  an  integrated/efficient  test  matrix. 

—Use  M&S  to  help  refine  and  focus  the  test  conditions 

—Identify  advanced  M&S  that  will  be  validated  as  part  of  the  test  process. 

—Concentrate  test  conditions  on  those  areas  with  high  programmatic  risks  (i.e.,  those  areas  with 
low  M&S  confidence  or  where  tests  results  are  least  predictable) 

—Integrate  OT&E  and  DT&E  test  requirements  as  much  as  feasible 

—Finalize  the  detailed  test  plan  and  conduct  a  formal  technical  review 
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-Identify  safety,  environmental,  and  other  test  constraints 

-See  Safety  section  within  Mission  Requirements/Systems  Integration  template 

-Prepare  a  detailed  systems  safety  analysis  with  supporting  documentation. 

-Think  through  the  test  step-by-step  and  look  for  environmental  impacts,  possible  failure 
modes,  and  document  them  (modify/tailor  test  plan  to  account  for  them) 

-Apply  for  all  environmental  permits,  explosive  site  plan  approvals,  waivers,  etc. 

Note:  Ensure  all  requirements  under  the  National  Environmental  Protection  Act  (NEPA)  have  been 
satisfied. 

-Identify  cost  and  schedule  constraints 

-Prepare  a  detailed  WBS  (identifying  tasks,  task  associated  manhour  and  material  costs,  and  contin¬ 
gencies)  and  integrated  detailed  schedule  (with  tracking  capabilities)  identifying  the  critical  path  and  task 
interactions. 

-Identify  instrumentation  and  data  requirements  as  early  as  possible 

—Using  the  integrated  test  requirements  document  and  the  developed  test  matrix  identify  the 
integrated  detailed  instrumentation  requirements  and  document. 

-Using  the  integrated  detailed  instrumentation  requirements  document  identify  the  integrated 
detailed  data  acquisition  system  requirements  document. 

—Using  the  integrated  detailed  data  acquisition  system  requirements  and  document  identify  the 
integrated  detailed  data  processing,  data  reduction  and  data  analysis  requirements. 

— Combine  these  requirements  into  a  Data  Analysis  Plan  (DAP) 

-Suggestion;  Draft  a  rough  outline  of  your  test  report  to  help  ensure  you  have  planned  for  ade¬ 
quate  instrumentation/data  requirements 

-Be  aware  of  data  sampling  considerations  (aliasing,  data  compression,  sample  rate,  nyquist 
criteria,  digital  or  analog  system,  etc.) 

-Ensure  instrumentation  system  is  common  airborne  instrumentation  system  (CAIS)  compati¬ 
ble  as  directed  by  Department  of  Defense  (DOD) 

—Identify  real-time  and  post-flight  data  requirements  and  cross-check  them  against  safety  plan 

-Develop  post-test  and  real-time  data  analysis/reduction  algorithms 

—Integrate  OT&E  and  DT&E  data/instrumentation  requirements  as  much  as  feasible 

-Determine  test  resources  needed  to  conduct  the  test  and  analyze  data 

-See  Resource  templates  and  Attachment  2  for  details 

-Rely  upon  statistical  analysis  tools  and  sound  engineering  judgment  to  minimize  test 
resources 

-Integrate  OT&E  and  DT&E  resource  requirements  as  much  as  feasible 
3.6.3.  Conduct  Test: 

-Plan  test  events 
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-Ensures  proposed  test  sequence  complies  with  1)  the  requirements  of  integrated  requirements 
document,  2)  the  integrated  matrix  3)  the  integrated  detailed  instrumentation  requirements  docu¬ 
ment,  4)  the  integrated  detailed  data  acquisition  system  requirements  document,  and  5)  the  integrated 
detailed  data  processing,  data  reduction  and  data  analysis  document. 

-Select  test  points  from  test  matrix 

—Ensure  scheduled  resources  are  in  place  to  support  test 

-Conduct  pre-test  briefings 

-Perform  test  and  collect  data 

-Gather  the  necessary  data  defined  in  the  test  plan  within  the  preplanned  test  methodology, 
safety,  environmental,  etc.,  constraints 

-Integrate  OT&E  and  DT&E  testing  as  much  as  feasible 

-Perform  real-time  data  quality  assessment 

-Minimizes  wasted  effort  (repeating  test  points  while  the  test  assets  are  in  place  and  the  test  is 
progressing  is  much  more  efficient  than  having  to  restart  the  test  at  a  later  date) 

-Test  personnel  should  have  a  thorough  understanding  of  the  SUT  (if  feasible,  system  design 
engineers  could  participate  in  test) 

-A  quick-look,  real-time  comparison  between  predicted  system  performance  and  measured 
performance  will  help  ascertain  if  test  points  need  repeating 

-Document  and  archive  test  events 

-Maintain  a  log  of  anomalies,  test  points  completed,  events  of  interest,  etc.,  for  historical  pur¬ 
poses  (the  log  helps  during  post-test  analysis) 

-Conduct  post-  test  reviews 

-Post-test  reviews  help  focus  post-test  data  analysis  as  well  as  help  scope  the  next  test 

3.6.4.  Perform  Post-Test  Analysis: 

-Determine  data  quality 

-Utilize  a  "quick-look"  data  processing  capability  to  ensure  data  quality  is  acceptable  before 
requesting  more  elaborate  processing 

-Perform  initial  assessment  of  system  performance 
-Process  data  into  engineering  formats 

-Focus  on  those  data  necessary  to  answer  the  test  objectives 
— Divide  data  into  time  slices  of  interest 
-Maintain  a  log  of  post-test  data  which  includes  parameters  such  as: 

— Time,  test  point  number,  test  conditions,  brief  explanation  of  test  points,  etc. 

-Compare  system  performance  to  predictions  and  system  requirements  (pre-test  analysis) 
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-If  results  are  different  than  expected  (use  engineering  judgment  and  experience)  then  perform 
the  following  (in  general  order): 

— Review  test  log  (from  conduct  test  step  above)  and  ensure  the  test  points  were  valid  and 
proper  test  procedures  were  followed 

—Verify  the  data  quality/accuracy  (instrumentation  calibrations,  data  decompression  algo¬ 
rithms) 

— Review  test  methodology  and  ensure  it  did  not  affect  data 
— Verify  data  analysis  algorithms  using  data  known  to  be  correct 

— Ascertain  if  prediction  tools  were  in  error  (if  data  is  determined  to  be  valid,  but  different 
than  predicted,  update  the  prediction  tool) 

— Ascertain  if  retest  is  necessary  (perform  only  as  last  resort)  through  engineering  judg¬ 
ment  and  consultation  with  other  personnel 

-Identify  deficiencies  in  test  item  or  test  procedure 

-If  test  procedure  is  in  error,  make  corrections  and  perform  retest  if  necessary 

—Provide  timely  feedback  to  the  appropriate  decision  makers  to  assist  them  in  determining  the 
need  to  go  to  the  Improve  Product  step. 

3.6.5.  Improve  Product: 

-Change  test  procedure  if  required 

—Eliminate  problem  sources  within  the  test  procedure 
-Provide  feedback  of  lessons  learned  to  test  personnel 
—Perform  retest  if  necessary 
-Determine  cause  of  deficiency 

—Once  identified,  provide  feedback  to  appropriate  personnel  for  timely  resolution 
-Improve  design  and  implement  change 

-Design  changes  can  take  a  substantial  amount  of  time  to  implement  and  may  require  a  signif¬ 
icant  amount  of  regression  testing 

-Maintain  configuration  control 

—If  multiple  design  changes  are  imple-mented,  a  structured  configuration  control  process  will 
be  critical  to  the  safe  and  successful  conduct  of  future  tests 

3.6.6.  Report  Results: 

-Document  test  results  (also  see  analysis  reports  section  of  Documents  template) 

—Final  reports,  interim  reports,  etc. 

-Keep  in  mind  that  final  reports  are  Historical  documents  that  can  significantly  benefit  the 
tester  for  future  test  programs 

-Document  test  item  deficiencies  or  recommended  enhancements 
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—See  DR  section  of  Documents  template 

-Provide  timely  feedback  to  decision  makers,  system  design  engineers,  program  managers,  other 
testers,  etc. 

-Use  appropriate  media  (account  for  coordination  time,  management  oversight,  etc.) 

— Daily  reports,  internal  memos.  E-mail,  telecon,  DRs,  etc. 

—Feedback  is  used  to  update  prediction  tools,  document  deficiencies,  make  system  changes,  or 
plan  retests 

—Ensure  personnel  who  need  the  test  results  receive  the  test  results. 


Figure  3.8.  Mission  Requirements  and  Systems  Integration. 
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3.7.  Mission  Requirements  and  Systems  Integration  Template: 

3.7.1.  Suitability,  (includes  reliability,  maintainability,  logistics  test,  availability,  supportability, 
compatibility,  transportability,  and  training)  is  the  degree  to  which  a  system  can  be  satisfactorily 
placed  in  field  use  to  accomplish  its  peacetime  and  wartime  mission 

-Guidance  contained  in  DODI  5000.2,  Parts  6&7,  AFI 10-602,  Determining  Logistics  Supportability 
and  Readiness  Requirements  (formerly  AFR  57-9),  AFI  20-101,  Instructions  for  Logistics  Strategic  Plan¬ 
ning  (formerly  AFR  400-1),  AFI  Air  Force  Aircraft  And  Equipment  Maintenance  (formerly  AFR 

66-1),  AFR  800-8,  Integrated  Logistic  Support  (ILS)  Program,  MIL-STD  1388-lA,  Logistic  Support 
Analysis,  and  Defense  Systems  Management  College  (DSMC)  Handbook,  Integrated  Logistics  Support 
Guide,  1st  Edition,  May  1986 

-Planning 

-SuitabiUty  is  as  important  as  system  effectiveness  (range,  speed,  survivability,  etc.) 

—Test  programs  must  implement  integrated  logistics  support  (ELS)  and  Logisitics  Test  (LOG 
TEST)  methodology  lAW  AFI  99-101  and  -102) 

—Suitability  must  be  designed  into  the  weapon  system  early  in  acquisition/  development  cycle 
and  cannot  be  "added"  as  an  after  the  fact 

—Weapon  system  "maintainers"  should  be  involved  as  early  as  possible  in  the  weapon  system’s 
acquisition/development  to  influence  the  design 
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—Ensure  software  suitability  is  tested 
-Requirements  Review 

-Review  ORD  to  ensure  suitability  related  ORD  requirements  are  complete  (for  example, 
clearly  define  the  downtime  assumptions  when  stating  availability  requirements) 

-Review  ORD  to  ensure  measurable  terms  are  used  when  stating  suitability  requirements 
(mean-time  between  critical  failures  for  reliability,  not  just  99  percent  reliable) 

-Ensure  software  suitability  is  sufficiently  addressed  in  requirements 

-Ensure  requirements  are  testable  (for  example,  99  percent  reliability  with  90  percent  confi¬ 
dence  and  with  one  failure  would  require  400  test  trials) 

-Test  Conduct 

-The  ALCs  depot  maintenance  infrastructure  resources  are  intended  to  support  sustainment 
activities  on  fielded  systems  not  development  or  acquisition  programs.  These  capabilities  can  and 
should  be  utilized  when  available  to  support  test  efforts  if  T&E  facilities  are  not  available  or  ade¬ 
quate  to  meet  certain  requirements. 

-"Piggy-back"  testing  as  much  as  possible  (use  onboard  BIT  data  if  available  as  well  as  routine 
ground  maintenance  data),  but  time  and  resources  (test  articles,  manpower,  etc.)  must  also  be  dedi¬ 
cated  for  suitability  testing  (for  example,  LOG  TEST) 

-Use  DR  system  lAW  TO  00-35D-54  to  not  only  document  deficiencies,  but  to  propose 
improvements/enhancements  to  the  system  (including  software) 

-Implement  a  shared  reliability,  maintainability,  and  availability  (RM&A)  database  controlled 
by  Joint  Reliability  and  Maintainability  Evaluation  Team  (JRMET)  -  see  API  99-101  and  -102 

-Data  Analysis 

-Data  are  reviewed/" authenticated"  by  JRMET  -  consists  of  DT&E,  OT&E,  SPO,  user,  and 
contractor  representatives 

-System  effectiveness  data  system  (SEDS)  is  used  to  facilitate  the  JRMET  process  by  storing 
and  processing  failure  and  maintenance  data  generated  during  flight  test 

-Should  be  interaction  between  JRMET  and  DR  Review  Board  (see  TO  00-35D-54,  USAF 
Deficiency  Reporting  and  Investigating  System,  for  DR  Review  Board  details) 

-OT&E  data  must  be  gathered/analyzed  in  accordance  with  (I AW)  OT&E  policy  (see  API 
99-102)  and  Public  law 

3.7.2.  Interoperability.  The  ability  of  weapon  systems  and  their  onboard  systems  to  transmit 
information/data  or  receive  information/data  effectively  from  other  weapon  systems/onboard  systems 

-Guidance 

—Department  of  Defense  Directive  (DODD)  4630.5,  Compatibility/lnteroperability/and  Inte¬ 
gration  ofCSI  Systems,  Department  of  Defense  Instruction  (DODI)e  4630.8,  Procedures  for  Com- 
patibility/Interopera-biity/and  Integration  of  C3 1  Systems,  and  Joint  Chiefs  of  Staff/Special 
Instruction  (JCS/SI)  6212.01  direct  what,  when,  and  how  systems  must  perform  interoperability 
T&E 
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— Interoperability  T&E  must  be  accomplished  prior  to  MS  III 

—The  Joint  Effectiveness  and  Interoperability  Office  (JEIO)  within  Defense  Information 
Systems  Agency  (DISA)  is  responsible  for  interoperability  testing  (see  JEIO  Circular  9002, 
Requirements  Assessments  and  Integration  Certification  of  C3I  and  AIS  Equipment  and  Sys¬ 
tems  for  guidelines) 

—The  Joint  Interoperability  Test  Center  (JITC),  Ft.  Huachuca  AZ  is  the  DOD  Executive 
Agent  for  interoperability  testing  and  certification  of  C4I  systems  and  equipment.  JITC 
observes  DT&E  and  OT&E  interoperability  test  events  or  conducts  inoperability  testing  when 
services  are  unable  to  address  interoperability  in  DT&E  and  OT&E. 

-Interoperability  is  a  Tri-Service  requirement 

-Planning 

-Primarily  a  C4I  function,  however  because  of  the  strong  linkage  between  C4I  and  avionics, 
interoperability  requirements  and  T&E  must  be  accounted  for  in  joint  A-P-A  and  C4I  related  T&E 

-Contact  C4I SFTC  Office,  Air  Force  Development  Test  Center  (AFDTC)/XRC,  Eglin 
AFB  FL  (DSN  872-2853)  for  further  information 

-SPO  must  ensure  resources  for  interoperability  T&E  are  included  in  program  planning  and 
documented  in  TEMP 

3.7.3.  SusceptibilityA^lnerability  Concerns.  Survivability  is  a  composite  of  susceptibility 
(probability  of  being  hit)  and  vulnerability  (probability  of  being  downed  by  a  hit).  Mathematically, 
Survivability  =  1  -  (Susceptibility  ¥  Vulnerability). 

-Guidance 

-For  assistance  in  implementing  a  survivability  program  for  your  weapon  system  contact 
AFMC  rep.  for  Joint  Technical  Coordinating  Group  on  Aircraft  Survivability  (JTCG/AS)  Office  at 
DSN  222-2120 

— DODI  5000.2,  Part  6,  sec.  F,  AFI 62-201,  Systems  Survivability,  and  MEL-STD  1799,  Surviv¬ 
ability,  Aeronautical  Systems  contain  survivability  guidance 

-Requirements  Review 

-Review  ORD,  and  system  threat  assessment  report  (STAR)  to  ensure  survivabihty  assump¬ 
tions/scenarios  are  adequately  defined 

-Planning 

—Ensure  TEMP  has  planned  for  resources  (threat  surrogates,  targets,  etc.)  and  their  availability 
to  conduct  survivability  testing 

—Ensure  testing  will  verify  and  validate  survivability  models 

-Testing 

—Survivability  is  not  one  number  (mueh  hke  there  is  no  such  thing  as  a  single  radar  cross  sec¬ 
tion  (RCS)  for  an  aircraft  because  it  is  dependent  upon  frequency,  look  angle,  polarization,  ete.),  but 
instead,  survivability  is  dependent  upon  many  factors,  including:  weapon  system's  concept  of  opera¬ 
tions  (CONOPS),  threat  type/density,  and  combat  scenario 
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— Therefore  because  of  the  large  number  of  tests  which  would  have  to  be  conducted  to 
assess  survivability,  modeling  and  simulation  (M&S)  must  be  used  to  assess  survivability 
(unless  very  narrow  constraints/assumptions  are  placed  on  survivability  requirement,  thereby 
drastically  reducing  the  test  matrix) 

-Susceptibility  and  Vulnerability  tests  are  used  to  verify  M&S  assumptions  and  results,  and  can 
also  be  used  to  directly  demonstrate  a  system’s  survivability  under  a  given  scenario 

-Individual  component  and  subsystem  vulnerabilities  (data  are  stored  in  a  vulnerability  data- 
base/budget  code)  are  aggregated  using  statistical  analysis  and  combined  with  shot-line  analysis 
(results  obtained  form  a  computer  model  of  aircraft)  to  calculate  an  aircraft’s  vulnerability  to  a  given 
hit 

— Component  and  subsystem  failure  modes  must  be  well  understood 

— Historical  data  from  similar  systems  live  fire  test  and  evaluation  (LFT&E)  should  sup¬ 
plement  your  aircraft’s  vulnerability  data 

— Testers  must  update  database/budget  code  using  LFT  data 

-The  susceptibility  of  a  weapon  system  is  derived  through  combining  a  series  of  probabil¬ 
ities  (i.e.,  probability  of  being  detected,  times  probability  once  detected  of  being  acquired,  times 
probability  of  being  targeted/lock-on  by  weapon,  times  probability  of  weapon  detonating  in 
vicinity  of  target  and  a  fragment  striking  aircraft) 

3.7.4.  Live  Fire  Test.  Title  10,  United  States  Code,  Chapter  139,  Section  2366,  et  seq.,  Congres- 
sionally  directs  live  fire  testing  and  further  guidance  is  contained  in  Secretary  of  Defense  Memoran¬ 
dum,  Live  Fire  Test  and  Evaluation  Guidelines,  27  Jan  94,  Department  of  Defense  Manual  (DODM) 
5000.2-M,  Parts  10  &  11,  and  AFI 99-105,  Live  Fire  Test.  Purpose  of  test  is  to  provide  timely  and  rea¬ 
sonable  assessment  of  the  vulnerability/lethality  of  a  system. 

-Guidance 

-Per  Section  2366  of  10  U.S.C.  139,  LFT&E  must  be  accomplished  sufficiently  early  in  a  pro¬ 
gram  to  identify  and  assess  possible  design  deficiencies  to  the  system  before  MS  HI 

-Typical  point  of  contention  is  whether  specific  weapon  system  is  "covered"  under  the  law  for 
LFT&E  (covered  system  is  defined  in  above  listed  documents,  but  open  to  interpretation.  Note:  may 

include  preplanned  product  improvement  (P^I)  programs) 

-Do  Not  assume  you  are  exempt  (i.e.,  not  a  covered  system),  instead  a  waiver  must  be  granted 
before  MS  II,  and  procedures  are  also  contained  in  above  listed  documents) 

-Beginning  with  component  level  T&E,  vulnerability  T&E  (may  include  live  fire)  continues 
through  EMD  with  additional  component/system  level  T&E,  and  progresses  to  LFT&E  of  produc¬ 
tion  representative  items  before  proceeding  beyond  low-rate  initial  production  (LRIP) 

—By  MS  I,  a  decision  should  be  made  whether  the  system  is  "covered"  and  initial  draft  LFT&E 
and  vulnerability/susceptibility  strategy  formulated.  By  MS  H,  the  TEMP  should  contain  a  mature 
strategy  (waiver,  or  full-up  LFT&E).  If  applicable,  entire  LFT&E  (including  reporting)  must  be 
completed  by  MS  DI 

-Planning 
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—Contact  Wright  Laboratories/FIV  for  assistance  in  developing  the  LFT&E  strategy 

-Perform  survivability  assessment  and  vulnerability  risk  versus  cost  assessment  to  ascertain 
LFT&E  cost  effectiveness  and  level  of  vulnerability  testing  (for  example,  a  weapon  system  with  an 
extremely  low  susceptibility  may  not  warrant  full-up  LFT&E) 

—Rely  extensively  on  modeling  and  simulation  to  assess  survivability  (composite  of  suscepti¬ 
bility  and  vulnerability)  and  for  planning  vulnerability  testing  shot-line  selection/test  matrix  (could 
include  live  fire) 

—Where  feasible,  use  component,  sub  scale,  or  partial  system  live  fire  tests  to  update  models, 
demonstrate  vulnerability  without  full-up  LFT&E,  or  in  buildup  to  full-up  LFT&E 

-Requirements  Review 

—Review  MNS,  ORD,  STAR,  and  COEA  to  plan  realistic  vulnerability  T&E  including  LFT&E 
against  realistic  threats 

—Include  LFT&E  and/or  survivability  assessment  strategy  (including  resources)  in  TEMP  lAW 
DODM5000.2-M,Part7 

-Test  Reporting 

—Document  LFT&E  results  and  submit  to  OSD  no  later  than  (NLT)  120  days  after  test 
completion  see  DODM  5000.2-M,  Part  10) 

3.7.5.  Climatic  Tests.  (Tests  conducted  in  the  following  environments:  icing,  extreme 
hot-weather/cold-weather,  blowing  sand/dust,  humidity,  vibration,  and  acoustic  to  assess  system  func¬ 
tionality  and  suitability) 

-Guidance  and  requirements  are  contained  in  MIL-STD  210,  Climatic  Extremes  for  Military  Sys¬ 
tems,  MEL-STD  810,  Environmental  Test  Methods,  and  AFR  80-31  “All-weather  Qualification  Program 
for  Aircraft  Systems  &  Material” 

-Planning 

-Ensure  climatic  T&E  is  included  in  TEMP  (allocate  time,  manpower,  and  resources) 

—Indoor  climatic  labs  (for  example,  the  McKinley  Climatic  Lab  at  Eglin  AFB  -  see  Attach¬ 
ment  2  for  information  regarding  climatic  labs)  allow  full-scale  weapon  system  testing  to  be  con¬ 
ducted  in  a  controlled  environment  (therefore  more  efficient  T&E  than  in  open-environment), 
however  the  aircraft  must  be  stationary  (engines,  APUs,  etc.,  may  be  running) 

—Plan  and  conduct  climatic  tests  during  deployment  to  extreme  weather  sites. 

-Controlled  in-flight  icing  may  be  conducted  using  an  instrumented  KC-135  aircraft  (contact 
412  TW/TSSS,  Edwards  AFB  CA,  DSN  525-9090  for  information)  or  for  slow  speed  in-flight  icing 
using  an  instrumented  CH-47  helicopter  (Contact  US  Army,  STEAT/AQ-CE,  Edwards  AFB  CA, 
DSN  527-3901) 

—Ensure  support  equipment  is  included  in  climatic  test  planning 

-Ensure  maintenance  related  tasks  are  evaluated  during  extreme  environment  testing 

—Component  climatic  requirement  T&E  are  performed  during  acceptance  and/or  qualification 
tests  (typically  at  contractor  facilities,  but  oversighted  by  government  quality  assurance  personnel) 
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-Incorporate  degraded  modes  into  climatic  test  plan  (perform  risk  versus  probability  of  occur¬ 
rence  analysis  to  assist  with  test  matrix  planning) 

-Requirements  Review 

-Review  component,  subsystem,  and  system  specifications  and  ORD  for  climatic  requirements 
for  adequacy  and  testability 

-Test  Conduct 

—Document  deficiencies  using  DRs  LAW  TO  00-35D-54 
-Verify  and  Validate  TO  maintenance  procedures 

-Assess  human  factor  related  tasks  in  realistic  climatic  conditions  (for  example,  maintenance 
tasks  with  cold  weather  gloves  on) 

3.7.6.  Human  Factors: 

-Guidance 

—Guidance  is  contained  in  DODI  5000.2,  Part  7-B,  and  DODM  5000.2-M,  Part  6-H 

-As  with  logistics  tests,  weapon  systems  must  have  good  human  factors  designed  into  them  and 
not  added  as  afterthought 

-Human  factors  have  a  significant  impact  on  mission  effectiveness  and  suitability 

-Human  factors  have  a  strong  influence  on  crew  station  layout  (member  of  cockpit  working 
group  per  API  63-112),  crew  visual  and  aural  annunciation,  and  mission  systems  (air  refueling,  cargo 
delivery,  night  vision,  ingress/egress  systems,  etc.),  but  must  also  be  involved  with  systems  which 
interface  with  personnel  (maintenance,  flying  qualities,  etc.) 

-Planning 

-Utilize  simulators  (with  required  fidelity)  as  much  as  possible  to  help  reduce/focus  more 
expensive  tests 

-Test  Performance  Assessment  and  Evaluation  System  (Test  PAES)  is  a  networked  computer 
system  available  for  assistance  in  human  factors  T&E.  Contact  412  TW/OG/DOEH,  DSN  525-9092 
for  information 

-Test  Conduct 

-Human  factors  tests  typically  piggy-back  on  systems  tests 

-Data  gathered  through  questionnaires,  video,  debrief  comments,  aircraft  instrumentation,  and 
aircrew  flight  reports 

—Document  deficiencies  with  DRs  lAW  TO  00-35D-54 

3.7.7.  Safety.  The  time  invested  into  a  well  planned,  well  thought  out  safety  plan  is  worth  the 
effort. 

-Guidance  is  contained  in  AFI 91-402,  USAF  Mishap  Prevention  Program  and  API  91-404 
-Planning 
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-Refer  to  your  individual  center’s,  lab’s,  agency’s,  etc.,  safety  office  for  detailed  safety  planning 
process  guidelines 

-Rely  upon  past  history/lessons  learned  for  effective  safety  planning  (each  center,  lab,  etc., 
should  have  a  safety  archive) 

-Use  failure  modes  and  effects  analysis  (FMEA)  and  similar  system  documents  to  aid  in  safety 
planning 

-Conduct  risk  versus  hazard  probability  assessment  and  if  applicable,  include  in  safety  plan 

-Allow  adequate  time  for  safety  plan  approval  cycle  in  test  scheduling 

-It  is  essential  for  effective  safety  planning  that  the  tester  know  the  SUT 

-Ensure  appropriate  real-time  instrumen-tation  required  for  safety  monitoring  is  included  in  the 
test  plan 

-Ensure  appropriate  safety  related  memo-randum  of  agreements  (MOAs),  treaties,  legal  docu¬ 
ments,  etc.,  are  in  place  prior  to  test 

-Ensure  the  test  plan  uses  a  risk  build  approach  to  testing  (i.e.,  progress  from  low  risk  tests  to 
progressively  higher  risk  tests  and  use  data  obtained  from  previous  tests  to  help  assess  risk) 

-Test  Conduct 

-Test  prebrief  should  be  mandatory  prior  to  all  tests 
— Brief  safety  constraints 

— Ensure  necessary  instrumentation  is  functioning 
— Gets  test  participants  focused  on  the  test 
-Ensure  safety  plan/constraints  are  adhered  to  during  test 

—If  a  test  team  is  conducting  the  test,  assign  a  test  director  with  safety  responsibility 
—Minimize  deviations  from  well  thought-out  safety  plan 
—Follow  mandated  safety  procedures  in  the  event  of  a  mishap  or  accident 
3.7.8.  Flight  Envelope  and  Flight  Manual  Development: 

-Testers,  SPO,  users,  and  contractor  must  work  together  to  determine  the  flight  envelopes  (permissi¬ 
ble,  operational,  etc.) 

—Review  ORD,  MNS,  Specifications,  and  flight  manuals  from  previous  aircraft  models  (i.e.  for 
F-15E  flight  envelope/manual  development,  use  F-15  C/D  data)  to  assess  envelope  requirements 

-M&S  should  be  the  primary  tool  to  assist  in  envelope  determination,  however  the  data  must  be 
validated  by  flight  test  and  flight  test  envelope  "comers"  should  be  flight  test  demonstrated  if  feasi¬ 
ble 

—If  your  program  involves  a  new  munitions  or  new  loading  on  existing  aircraft,  contact  the 
Armament/Munitions  Single  Face  To  Customer  (SFTC)  office  and/or  SEEK  EAGLE  office  at  DSN 
872-4190. 
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-The  SPO  is  responsible  for  the  TO-OOl,  Aircraft  Flight  Manual  development,  however  responsibil¬ 
ity  may  be  delegated  to  the  contractor  and  Responsible  Test  Organization  (RTO) 

-For  a  new  aircraft,  use  FMEA  and  flight  safety  related  documents  (for  example,  the  test  force 
should  develop  a  "what-if '  document  prior  to  first  flight  which  is  a  paper  analysis  of  possible  aircraft 
problems  and  procedures  for  in-flight  trouble  shooting,  aircrew  and  ground  emergency  procedures, 
and  recovery  procedures)  to  form  the  basis  for  preliminary  flight  manual  emergency  procedures 

-Use  the  test  force’s/contractor’s  aircraft  flight  operating  limitations  document  to  assist  in-flight 
manual  development 

3.7.9.  Aircraft-Stores  Certification  Program  (SEEK  EAGLE).  The  process  by  which  air- 
craft-store  configurations  are  certified  on  aircraft  to  meet  operational  requirements  specified  by  the 
using  command.  This  process  assures  aircraft-store  compatibility  (store  loading,  safe  carriage  and 
separation),  and  weapon  delivery  accuracy  verification. 

-Guidance 

— AFI 63-104,  The  SEEK  EAGLE  Program,  defines  the  SEEK  EAGLE  (SE)  process  and  the 
SEEK  EAGLE  Engineering/Test  Capabilities  Handbook  (Aug  1992)  identifies  the  location,  primary 
mission,  and  major  aircraft-store  certification  test  resources  available  at  government  test  and  analysis 
facihties.  Program  Management  Directive  for  Aircraft-Stores  Certification  (SEEK  EAGLE),  PMD 
5077,  provides  direction  and  funding  for  certification  projects. 

-Process 

-Begins  with  SE  Request  (SER)  submitted  by  the  using  conunand  or  the  Directorate  of  Interna¬ 
tional  Programs  (SAF/IAY)  for  Foreign  Military  Sales  International  programs  and  ends  with  the  pub¬ 
lication  of  related  technical  orders  and  user  acceptance  of  the  Operational  Flight  Program  accuracy, 
as  shown  in  Figure  3.7. 

-Each  SER  made  into  one  or  more  projects  designed  to  include  activities  required  to  complete 
certification  process.  Each  project  tailored  from  a  standard  template  which  considers  program 
requirements,  performance,  schedule,  cost  and  constraints. 

-SE  Management  Support  System  (SEMSS)  used  to  track  status  [Office  of  Primary  Responsi¬ 
bility  (OPR,  schedule,  cost,  etc.)]  for  each  activity. 

-Planning 

-SE  activities  integrated  with  acquisition  programs  early  in  life  cycle  (not  later  than  Demon- 
strationA'alidation  phase)  to  ensure  timely  consideration  of  realistic  operational  configurations  and 
to  integrate  SE  to  maximum  extent  possible  during  DT&E  and  lOT&E. 

-Certification  of  baseline  aircraft-store  configurations  completed  applying  Integrated  Weapon 
System  Management  (IWSM)  guidelines;  follow-on  certifications  accomplished  by  the  Air  Force 
Seek  Eagle  Office  (AFSEO). 

-Aircraft/Store  Integration 

-SE  not  responsible  for  development  or  modification  of  aircraft  or  store  to  achieve  actual  air¬ 
craft-store  configurations  requested  for  certification.  This  includes  avionics,  electrical  and  mechan¬ 
ical  integration  or  associated  modification  to  the  aircraft/store  to  provide  required  operational 
interfaces. 
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-Systems  integration  and  hardware  modification  are  the  responsibility  of  the  store  and/or  air¬ 
craft  System  Program  Offices. 
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Figure  3.9.  SEEK  EAGLE  Process. 
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Figure  3.10.  Documents. 
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3.8.  Documents  Template: 

3.8.1.  Acquisition/Requirements  Documents: 
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-The  Operational  Requirements  Document  (ORD)  is  the  responsibility  of  the  user  and  documents 
how  a  system  will  be  operated,  deployed,  employed,  and  supported 

“ORD  format/content  guidance  is  contained  in  DODM  5000.2-M,  Part  3,  and  API  10-601 

—Ensure  the  requirements  are  realistic,  testable  and  are  expressed  in  measurable  terms  (range, 
endurance,  functionality,  reliability,  etc.) 

“Provides  tester  with  system  CONOPS  and  sunamarizes  the  threats  (details  in  STAR) 

—Testers  should  use  it  for  planning  operationally  realistic  tests  (including  DT&E) 

-Cost  and  operational  effectiveness  analysis  (COEA)  is  the  responsibility  of  the  user  and  is  a  quanti¬ 
tative  analysis  which  identifies  the  preferred  system 

—Guidance  is  contained  in  DODI  5000.2,  Part  4-E  and  DODM  5000.2-M,  Part  8 

—Documents  the  Measure  of  Effectiveness  (MOE)  for  determining  whether  the  system  meets 
the  users  needs  and  as  such,  should  be  reviewed  for  testability  and  to  determine  whether  assumptions 
and  models  are  valid 

—Provides  testers  (OT&E  and  DT&E)  with  the  critical  operational  issues  (COIs),  their  MOE, 
and  costs  (should  be  used  when  making  T&E  resource  planning  decisions) 

—If  the  COIs  or  MOE  are  not  directly  testable,  lower  level  test  criteria.  Measure  of  Performance 
(MOP)  -  speed,  range,  handling  qualities,  etc.,  must  be  developed 

—Assesses  sensitivity  of  critical  performance  alternatives 

—Testers  should  ensure  COEA  key  assumptions  and  estimates  are  testable 

-System  Maturity  Matrix  (SMM)  is  the  responsibility  of  the  SPO  and  is  an  acquisition  management 
tool  used  for  evaluating  weapon  system  capability  versus  time  and  defines  criteria  for  risk  management 
and  analysis 

—See  OSD  Policy  92M-001  for  guidance  regarding  SMM  application 

-Time-phased  comparison  of  the  user’s  system  requirements  and  contractual  specifications 
which  lead  to  maturity 

—Includes  significant  parameters  necessary  to  measure  progress  towards  meeting  the  user’s 
needs 

—Ensure  requirement  criteria  traceability  to  the  ORD 

—Establishes  system  development  exit  criteria  (see  Exit  Criteria  template  for  additional  infor¬ 
mation) 

-Integrated  logistics  support  plan  (ILSP)  is  the  responsibility  of  the  SPO  (Logistics  Manager)  and 
provides  ILS  information  used  to  develop  and  define  supportability  design  factors  and  develop  integrated 
system  support  structure.  Identify  and  establish  requirements  for:  maintenance  concept,  manpower 
(skills,  numbers  of,  etc.),  supply  support,  support  equipment,  training,  packaging-handling-storage-and 
transportation,  technical  data  (TOs),  and  system  support  facilities 

—Guidance  contained  in  DODI  5000.2,  Part  7,  API  10-602,  API  20-101,  API  21-101,  APR 
800-8,  MEL-STD  1388-lA,  and  DSMC  Handbook,  Integrated  Logistics  Support  Guide,  1st  Edition, 
May  1986 
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-Documents  the  Integrated  Weapon  System  Management  (IWSM)  approach  to  system  fielding 

-Integrity  Program  documents  are  Military  Standards  that  provide  general  requirements  and  specific 
tasks  to  achieve  software,  aircraft  structural,  engine  structural,  avionics  and  mechanical  integrity  during 
the  development  and  deployment  of  systems  and  equipment.  These  documents  impose  a  requirement  for 
a  Development  Integrity  Program  with  an  objective  to  assure  that  operational  needs  are  met,  the  system  is 
supportable  and  can  be  developed  within  schedule  and  resource  constraints  of  the  acquisition  program. 
The  test  requirements  to  verify  the  design  and  performance  are  generated  as  Section  4  requirements  in 
these  documents.  The  integrity  documents  include: 


-MIL-STD  1803 
-MIL-STD  1530 
-MIL-STD  1783 
-MIL-STD  1796 


Software  Development  Integrity  Program 
Aircraft  Structural  Integrity  Program 
Engine  Structural  Integrity  Program 
Avionics  Integrity  Program 


-MIL-STD  1798  Mechanical  Equipment  and  Subsystem  Integrity  Program 
3.8.2.  System  Configuration/Engineering  Documents: 


-Baseline  documentation  (functional,  allocated,  product,  process  baselines)/(A,  B,  C,  D,  or  E  type 
specifications).  These  documents  identify  system  requirements  at  progressively  lower  levels  and  contrac¬ 
tually  bind  the  contractor 


-Guidance  contained  in  DODI  5000.2,  Part  9-A,  MIL-STD  480,  MIL-STD  483,  APR  800-60, 
and  DSMC  Handbook,  Systems  Engineering  Management  Guide,  2nd  Edition,  1990 

-The  various  specification  levels  provide  a  clear  audit  trail  beginning  with  user  requirements  to 
product/item  requirements 

-Formally  reviewed  at  various  government  reviews  [preliminary  design  review  (PDR),  critical 
design  review  (CDR),  system  requirements  review  (SRR),  etc.]  dependent  upon  acquisition  phase 

-Ensure  the  specifications  are  relevant  and  do  not  Just  "gold  plate"  the  user’s  requirements 

—Ensure  the  requirements  are  testable 

-Interface  Control  Documents  (ICDs)  control  interfaces  between  various  components,  subsystems, 
or  systems  by  documenting  interface  agreements,  architecture,  and  requirements  (timing,  bit-map,  proto¬ 
col,  etc.) 


-Written  by  system  designer(s)  (usually  the  contractor) 

-Essential  document  to  control  software-to-software  interfaces  as  well  as  software-to-hardware 
interfaces 


—Critical  document  during  problem  troubleshooting 

-Configuration  documents  are  constantly  changing  documents  which  evolve  with  the  system.  They 
document  the  exact  functional  and  physical  characteristics  of  the  system  versus  time.  The  documents  are 
at  various  levels  depending  upon  the  level  of  detail  needed  to  identify  a  configuration  and  contain  infor¬ 
mation  such  as  component  (hardware  and  software)  serial  numbers,  versions,  modifications,  maintenance 
write-ups,  and  instrumentation  configuration 

-System  detailed  characteristics  are  documented  by  the  designer  (usually  the  contractor) 
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—The  system  configuration  is  controlled  through  the  Configuration  Control  Board  (CCB)  made 
up  of  representatives  from  DT&E,  OT&E,  contractor,  SPO,  and  sometimes  the  user 

-These  are  essential  documents  to  the  tester  in  that  they  define  exactly  what  configuration  is/ 
has  been  tested 

—Tester  must  ensure  the  configuration  is  valid  for  the  test  to  be  performed 
3.8.3.  Test  Documents: 

-The  TEMP  is  the  responsibility  of  SPO  and  is  prepared  as  early  in  the  acquisition  process  as  possi¬ 
ble.  Identifies  and  integrates  high  level  T&E  objectives,  responsibilities,  resources,  and  schedules  in  sup¬ 
port  of  key  program  decision  points 

—Guidance  is  contained  in  DODI  5000.2,  Part  8,  and  DODM  5000. 2-M,  Part  7 

-Written  through  Test  Plan  Working  Group  (TPWG)  forum  and  is  a  product  of  the  T&E  plan¬ 
ning  process 

—Top-level  summary  of  the  program’s  T&E  process  implementation 
—Used  to  generate  lower-level  detailed  T&E  plans 

—Required  at  milestone  I  and  updated  at  each  major  milestone  or  significant  change  in  the  status 
of  the  acquisition  program 

—Ensure  the  plan  is  executable,  defendable,  and  uses  a  building  block  approach  for  test  progres¬ 
sion  (see  2.6.8.  for  further  details) 

—Ensure  traceability  to  requirements  documents  (ORD,  COEA,  STAR,  ILSP,  etc.) 

—Tailor  each  version  to  support  appropriate  milestone  decision 

—Include  both  hardware  and  software  test  information 

-Detailed  test  plans  flow  top-down  from  TEMP  and  document  test  planning  to  a  level  detailed 
enough  to  perform  a  specific  test  and  gather  required  information.  Guidance  is  contained  in  AFI 99-101 

—The  format  and  content  for  detailed  test  plans  vary  and  are  usually  directed  by  individual  cen¬ 
ters,  agencies,  and  labs 

-Detailed  test  plans  should  be  written  and  coordinated  among  all  test  participants  (DT&E, 
OT&E,  contractor,  and  SPO)  to  maximize  test  efficiency  (for  example  DT&E  and  OT&E  should 
strive  to  use  the  same  instrumentation,  data,  test  points,  etc.) 

—Ensure  tests  can  be  conducted  within  safety,  environmental,  etc. 

—Perform  any  necessary  hazard  (safety,  environmental,  etc.)  analysis  prior  to  conducting  test 
and  ensure  results  are  utilized  in  detailed  test  planning 

-Deficiency  Reports  (DRs)  [formerly  service  reports  (SRs)]  are  government  written  and  controlled 
documents  which  identify  system  (hardware  and  software)  deficiencies 

—Of  paramount  importance  to  testers  and  decision  makers  by  providing  feedback  into  the  system 
engineering  process  to  make  corrections 

—Essential  for  correcting  deficiencies  early  (recommend  being  implemented  during  contractor 
DT&E  using  watch  items  in  conjunction  with  formal  DRs-  see  TO  00-35D-54) 
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-Directed  and  Implemented  lAW  TO  00-35D-54  during  government  testing 

-Prior  to  government  conducted  testing,  a  contractor-owned  and  -  based  DR  system  must  be  estab¬ 
lished 

— DRs/watch  items  are  reviewed,  approved,  and  prioritized  through  DR  Review  Boards  chaired  by 
testers 

-Resolution  of  DRs  is  accomplished  through  material  improvement  project  (MIP)  chaired  by  SPO 

-Analysis  Reports  (interim,  final,  daily,  etc.)  take-on  many  forms  dependent  upon  the  time  span  cov¬ 
ered  for  a  given  (or  set  of)  test,  the  testing  agency/command,  and  the  report  recipient 

—Ensure  reports  are  timely  for  decision  makers  and  "answer  the  mail" 

-Formats,  content,  etc.  are  directed  by  individual  centers,  agencies,  etc. 

-It  is  helpful  to  outline  the  test  reports  prior  to  testing  to  ensure  necessary  data  is  gathered  for 
decision  makers 

—Keep  in  mind  that  final  reports  are  historical  documents  and  therefore,  must  adequately 
describe  the  system  configuration,  data  analysis,  assumptions,  etc. 


Figure  3.11.  Embedded  Software. 
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3.9.  Embedded  Software  Template: 

3.9.1.  Guidance: 

-Embedded  software  testing  encompasses  a  wide  range  of  complex  areas  from  initial  software  test¬ 
ing  at  the  module  level  (DOD-Std-2167,  Defense  System  Software  Development)  to  integrated  DT&E  and 
OT&E  testing.  For  the  purposes  of  this  manual  it  is  impossible  to  cover  all  aspects  of  the  embedded  soft¬ 
ware  testing  process.  Therefore,  the  generic  concept  of  testing  embedded  software  will  be  covered  with 
the  following  documents  recommended  as  sources  of  information  on  the  software  development  and  test¬ 
ing  processes. 

-Embedded  software  guidance  is  contained  in  Secretary  of  the  Air  Force  Memorandum,  Guidance 
for  the  Successful  Acquisition  of  Computer  Dominated  Systems  and  Major  Software  Development, 
DOD-STD-2167,  DOD-STD-2168,  Defense  System  Software  Quality  Program,  AFR  800-14,  Life  Cycle 
Management  of  Computer  Resources  in  Systems,  AFLCR  800-21,  AFMC  Pamphlet  800-45,  Risk  Abate- 
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merit,  AFMC  Pamphlet  800-60,  IWSM  Guide,  AFMC  Embedded  Software  Management  Plan,  1  Sep  93, 
and  MIL-STD-1803,  Software  Development  Integrity  Program,  15  Dec  1988. 

-The  extent  of  embedded  software  testing  is  determined  by  the  level  of  testing  required.  Test  team 
involvement  should  start  at  the  earliest  possible  stage  of  the  project.  Ideally  the  test  team  would  define  the 
high  level  test  criteria  during  the  concept  exploration  phase,  participate  in  generating  the  TEMP  and  par¬ 
ticipate  with  the  Computer  Resources  Working  Group  (CRWG). 

-Software  testing  starts  at  the  computer  software  unit  (CSU)  level  followed  by  the  computer  software 
component  (CSC)  and  computer  software  configuration  item  (CSCI)  testing.  Normally  this  testing 
ensures  that  the  intended  software  functionality  has  been  achieved.  The  next  level  (normally  a  system 
level  test)  tests  the  CSCIs  integration  with  the  hardware  and  other  CSCIs  in  an  operational  environment. 
This  test  is  intended  to  "drive  out"  possible  operational  problems.  Each  level  is  increasingly  complex  and 
requires  understanding  of  the  software  and  software  environment  being  tested.  The  level  at  which  the 
embedded  software  is  being  tested  determines  which  of  the  steps  into  this  template  will  be  used.  This 
determination  is  made  by  the  test  team  who  is  ultimately  responsible  for  defining  and  conducting  the  test. 

3.9.2.  Determine  Test  Objectives: 

-Review  requirements  to  ensure  that  each  section  3  (specification)  requirement  has  a  corresponding 
section  4  paragraph  stating  how  that  requirement  will  be  tested 

-Review  Requirements  Document 

-Appropriate  DOD-Std-2167A  documents  [System  Requirements  Specification  (SRS),  Sys¬ 
tem/Segment  Specification  (SSS),  System/Segment  Design  Document  (SSDD),  System  Design 

Document  (SDD),  Software  Test  Plan  (STP),  Standard  (STD),  Interface  Design  Document  (IDD)] 

-Operational  Requirements  Document  (ORD),  Mission  Needs  Statement  (MNS) 

-Identify  purpose  of  test 

-To  test  functionality,  basic  or  expanded 

—To  write  or  verify  documentation  (TOs,  manuals) 

-Formulate  test  plan  (see  DOD  5000  series  and  the  Exit  Criteria  template  for  guidance) 

-Identify  test  completion  criteria 

—How  will  you  know  when  the  test  is  complete? 

—What  is  success?  What  will  mean  re-test  is  required? 

-Identify  test  result  reporting  requirements 
—Who  will  need  the  results? 

-What  data  formats  and  media  are  required? 

—Prototype  the  results:  draft  a  rough  outline  of  the  test  report  early  and  verify  with  the 
intended  receivers  that  the  proposed  data  and  formats  will  meet  their  needs 

3.9.3.  Conduct  Pre-Test  Analysis: 

-Predict  software  performance 
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—Review  previous  test,  if  any,  or  similar  software  systems  data  to  help  predict  software  perfor¬ 
mance 

—Use  engineering  judgment  or  past  experience 

-Use  modeling  and  simulation  to  help  predict  performance 

-Using  software  listings  and  documentation  to  guide  you,  determine  software  design  limits  and 
predict  performance  at  and  beyond  those  limits 

-Identify  test  conditions 

-Where  will  testing  occur?  In  isolation  or  attached  to  the  rest  of  the  system? 

-On  the  ground,  in  the  air,  local  or  remote,  special  facility? 

-Develop  test  matrix,  using  modeling  and  simulation,  statistical  analysis  and  good  judgment  to 
focus  the  test  on  areas  where  confidence  is  low  or  risk  of  failure  is  high 

-Test  matrix  should  exceed  software  design  limits  as  determined  by  examination  of  code  list¬ 
ings  and  software  documentation 

-Test  to  find  errors,  and  also  show  what  works;  each  bug  found  and  eliminated  increases  the 
value  of  the  software 

-Identify  safety  and  environmental  test  con-straints 

—Identify  local  safety  hazards  caused  by  test  and  incidental  to  test  , 

— High  or  low  temperatures,  wet,  slippery,  freezing,  caustic,  poisonous 

— Know  what  other  software,  systems  or  subsystems  could  be  affected  or  activated  by  the 
test,  and  possible  safety  hazards  as  a  result:  e.g.,  testing  fire  detection  subsystem  could  activate 
fire  suppression  subsystem 

— Identify  environmental  requirements,  such  as  anechoic  or  thermal 
—Identify  possible  environmental  hazards 
— Explosion,  fire,  poisonous  gas 
-Determine  instrumentation  and  data  require-ments  early 

—What  do  you  want  to  report?  What  do  you  need  to  measure? 

-Review  the  prototype  results  report  (see  Determine  Test  Objectives,  on  this  template  to 
ensure  that  the  expected  results  are  linked  to  the  necessary  instrumentation) 

-Make  sure  you  know  what  you  are  really  measuring 

—Will  you  need  and  will  the  software  provide  "diagnostics"  if  requested? 

-Determine  Test  Resources  needed  to  conduct  test  and  analyze  data 

-People,  fixtures,  facilities,  analysis  software 

3.9.4.  Conduct  Test: 

-Collect  data 

—Follow  test  plan  and  collect  data 
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-Testing  should  be  scripted,  then  follow  the  script 
-Integrate  OT&E  and  DT&E  testing  whenever  possible 
-Perform  real-time  data  quality  assessment 

-Use  quick-look  capabilities  where  available,  devise  your  own  when  not 

-Much  cheaper  to  repeat  a  test  point  on  the  spot,  when  everything  is  in  place,  rather  than  having 
to  set  it  all  up  again  later 

-Compare  predicted  with  actual  to  determine  if  test  points  ought  to  be  repeated 

—Test  personnel  should  have  good  understanding  of  the  embedded  software  in  the  SUT.  If  pos¬ 
sible,  involve  software  designers/developers  in  the  testing 

-Document  test  events 

-Testing  should  be  scripted,  and  a  log  updated  as  each  test  point  is  completed 
—Note  all  anomalies  and  anything  interesting,  especially  unexpected  events 
-More  is  better;  good  documentation  will  help  the  post-test  analysis 
-Conduct  pre-  and  post-test  reviews 

—Pre-test  briefings  will  help  focus  personnel  on  the  task  at  hand 
—Use  check-off  list  to  ensure  nothing  is  missing 
—Refreshes  testers  on  safety,  test  and  environmental  constraints,  hazards 
-Post-test  reviews  help  focus  analysis  and  help  scope  the  next  test 

3.9.5.  Perform  Post-Test  Analysis: 

-Compare  measured  data  to  predicted  data  and  software  requirements  (pre-test  analysis) 

-If  different  more  than  expected  (use  engineering  judgment  and  experience)  then  perform  the  follow¬ 
ing  (in  general  order) 

-Review  test  log  and  ensure  the  test  point  was  valid 
—Verify  the  data  quality/accuracy 

—Review  test  methodology  and  ensure  that  it  did  not  affect  the  data 
-Verify  data  analysis  algorithms  using  data  known  to  be  correct 
-Ascertain  if  prediction  tools  were  in  error 
-Ascertain  if  retest  is  necessary 
-Identify  deficiencies  in  the  software  or  procedure 

-If  test  procedure  is  in  error,  make  corrections  and  perform  retest  if  necessary 

3.9.6.  Improve  Product: 

-Change  test  procedure  if  required 

—Correct  problem  sources  within  the  test  procedixre 
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—Perform  retest  if  necessary 
-Determine  cause  of  deficiency 

-Identify  and  correct  problems  as  early  as  possible  during  the  software  development  process 

-Correcting  a  software  error  during  the  design  phase  is  substantially  less  costly  than  discover¬ 
ing  an  error  in  the  production  system 

-Improve  design  and  implement  change 

-Before  proceeding  through  the  test  process  steps,  it  may  be  necessary  to  access  and  possibly 
revise  the  system  test  objectives 

-Modification  of  the  system  architecture  may  require  a  significant  amount  of  regression  testing 
-Maintain  configuration  control 

-A  structured,  well  defined  configuration  control  process  will  significantly  reduce  both  soft¬ 
ware  and  system  risks  during  development 

3.9.7.  Report  Results: 

-Document  the  software  deficiencies  and/or  recommended  enhancements 
-See  DR  section  of  the  Documents  template 

-Provide  timely  feedback  to  decision  makers,  system  designers,  program  managers,  other  testers,  etc. 
—Use  appropriate  media  (account  for  coordination  time,  management  oversight,  etc. 

— Daily  reports,  internal  memos.  E-mail,  telecon,  DRs,  etc. 

-Feedback  is  used  to  update  prediction  tools,  document  deficiencies,  make  software  changes, 
or  plan  retests 

-Document  test  results 

—Final  reports,  interim  reports,  etc. 

Figure  3.12.  Exit  Criteria. 


Integrated  System  Development  Phase 
Fielded  System  Testing 
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3.10.  Exit  Criteria  Template: 

In  planning  and  executing  a  test  program,  it  is  important  to  understand  and  define  criteria  which  identify 
the  successful  completion  of  test  phases  and  individual  tests.  The  following  templates  are  suggestions/ 
things  to  consider  when  identifying  exit  criteria  during  the  various  stages  of  a  weapon  system’s  develop¬ 
ment. 

3.10.1.  General: 

-The  weapon  system’s  System  Maturity  Matrix  (SMM)  is  a  tool  used  to  track  technical  progress 
against  the  program’s  critical  system  characteristics  (see  DODI  5000.2  for  definition),  allocated  require¬ 
ments  (system  specifications)  and  Air  Force  Acquisition  Policy  92M-001 

-The  SMM  also  defines  exit  criteria  for  completing  each  acquisition  phase/MS 

-See  Documents  template  for  additional  information  regarding  SMMs  and  system  specifica¬ 
tions 

-The  tester  should  develop  criteria  which  identify  the  completion/success  criteria  for  critical  aspects 
of  the  development  process 

-For  example,  delivery  of  a  specific  set  of  ICDs  may  be  critical  before  proceeding  through  a 
given  subsystem’s  development 

-In  short,  the  testers  in  conjunction  with  the  SPO,  should  identify  the  critical  events,  system  perfor¬ 
mance,  test  prerequisites,  documents,  data,  and  resources  needed  to  minimize  risks  and  proceed  effi¬ 
ciently  through  each  stage  of  a  test  program 

3.10.2.  Concept/Early  Planning 
-Test  program  planning 

-Good  planning  is  always  the  key  to  good  testing  and  good  planning  always  starts  with  review¬ 
ing  requirements 

— See  Documents  template  for  suggestions  on  reviewing  requirement  documents 

—The  results  of  COEA  and  models  should  be  reviewed  to  identify  the  key  drivers  in  sys¬ 
tem  performance  and  identify  sensitive  parameters 

— Review  the  various  baselines  (functional,  allocated,  etc.)  to  identify  desired  component, 
sub-system,  and  system  performance 

— Review  development  schedules,  deliverables,  and  contract  data  requirements  list 
(CDRL)  items  to  ensure  prerequisite  tests,  data,  support  equipment,  and  test  articles  are  prop¬ 
erly  identified  and  can  be  delivered  within  the  contract  constraints 

-Develop  a  test  program  built  upon  answering  the  mail  on  all  the  requirements  discussed  above 

— The  TEMP,  SMM,  program  introduction  document  (PID)/statement  of  capability 
(SOC),  detailed  test  plans,  etc.,  should  document  a  clear  audit  trail  from  weapon  system 
requirements,  to  test  objectives,  to  individual  test  success  criteria 

— The  TEMP,  SMM,  PID/SOC,  detailed  test  plans,  etc.  should  also  baseline  an  executable 
test  pro-gram  with  measurable  exit  criteria 
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— Ensure  test  strategies  have  flowcharts  which  identify  exit  criteria  and  include  adequate 
(numbers  of  and  timing  of)  program  reviews 

— Develop  a  program  which  allows  early  identification  and  correction  of  problems  on 
those  areas  identified  above  to  be  risky 

—Review  USAF/TE’s  Certification  of  Readiness  for  Dedicated  OT&E  Templates  and 
ensure  the  test  program  will  give  DT&E  enough  data  and  sufficient  confidence  when  the 
time  comes  to  certify  readiness  for  dedicated  OT&E 

—Failure  to  meet  a  given  exit  criteria  will  require  either  additional  testing  and/or  a  reassessment 
of  the  risk  and  performance  requirements. 

3.10.3.  Component  Development  Phase: 

-Testing  is  conducted  to  evaluate  how  well  the  components  perform  within  their  respective  sub¬ 
systems  as  designed  and  whether  they  meet  specifications/baselines  (generally  component  testing  is  too 
low  level  to  assess  against  user  requirements) 

—Typically,  the  component  under  test  is  stimulated  using  inputs  derived  from  an  external  simu¬ 
lation  (hardware  and/or  software) 

-Ensure  the  component’s  software  is  adequately  exercised  (adequately  depends  upon  the  code’s 
size,  complexity,  number  of  interfaces,  etc.) 

-Typical  component  exit  criteria  include:  Meeting  environmental  requirements  (strength,  life, 
weight,  temperature,vibration,  etc.),  airworthiness  certification,  component  level  software  module  per¬ 
forms  as  required,  subsystem  simulation  in  which  component  is  installed  meets  requirements,  updating  a 
model  from  component  test  results,  or  verifying  a  design  change 

-Usually  for  a  new  weapon  system  program,  component  level  testing  is  conducted  primarily  using 
Measurement  Facilities  (MFs),  System  Integration  Laboratories  (SILs),  and  M&S  facilities 

—See  Resource  templates  and  Attachment  3  for  additional  information 

3.10.4.  Subsystem  Development  Phase: 

-Testing  is  conducted  to  evaluate  how  well  a  given  subsystem  performs  against  design  requirements 
internal  to  its  respective  system  and  how  well  it  interfaces  with  other  subsystems  (internal  and  external  to 
its  system) 

-Subsystem  testing  is  typically  the  first  opportunity  to  test  at  a  level  near  the  user  requirements 
level  without  excessive  extrapolation 

-Ensure  component  and  subsystem  ICDs  are  written  and  adhered  to  before  proceeding  too  far 
into  system  level  development  ("too  far"  depends  upon  risk,  adequate  schedule  allowance  to  correct 
problems,  resource  availability,  etc.) 

-Ascertain  what  capability  is  needed  and  demonstrated  from  the  subsystem  before  progressing 
too  far  into  system  level  development 

-Typical  exit  criteria  include:  updating  a  system  or  subsystem  level  model  from  subsystem  test 
results,  demonstrating  redundancy  requirements,  demonstrate  the  subsystem’s  functional  requirements, 
meeting  life-cycle  endurance  requirements  (x  number  of  cycles  before  failure),  etc. 
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-Usually  for  a  new  weapon  system  program,  subsystem  testing  is  conducted  primarily  using  SELs, 
Hardware-In-The-Loop  (HITL),  and  M&S 

-See  Resource  templates  and  Attachment  3  for  additional  information 

3.10.5.  System  Development  Phase: 

-Testing  is  conducted  to  ensure  the  air  vehicle  is  safe,  evaluate  how  well  the  system  performs  against 
design  requirements,  and  obtain  a  preliminary  assessment  of  the  weapon  system’s  operational  effective¬ 
ness  and  suitability  (to  support  certification  of  readiness  for  dedicated  OT&E)  before  proceeding  into  the 
integrated  system  testing  phase 

-Ascertain  air  vehicle’s  airworthiness  prior  to  first  flight 

— Verify  installed  component,  subsystem,  and  system  structural  integrity  and  functionality 

— Assess  performance  capability 

— Verify  system  stability 

—Verify  subsystem/system  compatibility  [Electromagnetic  Interference  (EMI),  Electro¬ 
magnetic  Compatibility  (EMC),  etc.) 

— If  sufficient  flight  risk,  perform  taxi  tests 
—Identify  test  point  success  criteria  required  for  safe  envelope  expansion 

—Ensure  test  points  adequately  test  system  functionality  and  gain  confidence  before  pro¬ 
ceeding  to  successive  test  points 

-Once  the  system  is  mature  enough  (SPO,  DT&E,  and  OT&E  discretion),  ensure  DT&E  tests 
are  operationally  realistic  and  adequately  address  operational  requirements  (COIs,  MOE,  and  MOPs) 

— Refer  to  USAF/TE’s  Certification  of  Readiness  For  Dedicated  OT&E  Templates  for 
details 

-Typical  exit  criteria  include:  demonstrating  basic  airworthiness  tasks  (takeoff,  landing,  navigating, 
refueling,  etc.),  demonstrating  preliminary  mission  level  tasks  (bombing,  aerial  delivery,  SEAD,  etc.), 
evaluating  maintenance  tasks,  technical  order  validation,  developing  the  flight  manual  to  a  pre-deter- 
mined  level.  Certification  of  Readiness  for  Dedicated  OT&E,  etc. 

-System  level  testing  is  primarily  conducted  in  ISTFs  and  OARs 

-See  Resource  templates  and  Attachment  3  for  additional  information 

3.10.6.  Integrated  System  Development  Phase: 

-Testing  is  conducted  to  evaluate  the  weapon  system’s  operational  effectiveness  and  suitability  in  an 
environment  which  as  close  as  practical  replicates  the  weapons  system’s  intended  operational  environ¬ 
ment 

—DT&E  test  objectives  should  concentrate  on  conducting  operationally  realistic  tests  which 
adequately  stimulate  and  evaluate  the  weapon  system  in  support  of  dedicated  OT&E  certification  (if 
not  completed  in  previous  phase) 

—OT&E  testers  must  ensure  adequate  data  is  gathered  to  perform  system  evaluation  to  answer 
all  COIs  (require  MOE  and  MOP  evaluation/rollup) 
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-Ensure  DRs  are  closed  or  in  MIP  process  (see  TO  00-35D-54  for  details)  prior  to  fielding 

-Typical  exit  criteria  include:  demonstrating  key  parameter  thresholds  in  ORD  (see  API  10-601  and 
DOOM  5000.2-M,  Part  3),  answering  COI,  MOP  criteria,  providing  OT&E  final  report  to  OSD,  collect¬ 
ing  data  for  high  level  model  verification,  validation  and  accreditation  ( VV&  A),  etc. 

-System  level  testing  is  primarily  conducted  in  OARs  and  is  supported  by  M&S 

-See  Resource  templates  and  Attachment  3  for  further  information 

3.10.7.  Fielded  System  Testing: 

-P^I  and  upgrade  programs  (weapon  system  is  out  of  production)  require  some  level  of  T&E.  The 
level  required  is  very  dependent  upon  the  exact  nature  of  the  mod/upgrade.  Some  may  require  only 
FOT&E,  but  others  may  require  a  full  blown  DT&E/OT&E  (in  which  case  the  acquisition  and  develop¬ 
ment  cycle  starts  over) 

—Testing  is  conducted  to  ensure  the  mod/upgrade  meet  requirements  (user  and  in  some  cases, 
detailed  specifications) 

-Typical  exit  criteria  include:  verifying  weapon  system  increased  suitability  and/or  effectiveness 
(OT&E  exit  criteria),  evaluating  changes  to  system  for  effects  on  safety,  flight  characteristics  for  flight 
manual  changes,  increased  system  performance 

-Fielded  system  testing  is  primarily  conducted  on  OARs  or  ALCs  (see  resource  templates  and 
Attachment  3  for  further  information). 

3.11.  Airframe  Mission  Area  Template; 

Test  Resources  (Also  see  Attachment  3,  T&E  Resources,  for  details) 

3.12.  Propulsion  Mission  Area  Template: 

Propulsion  Test  Resources  (Also  see  Attachment  3,  T&E  Resources,  for  details) 

3.13.  Avionics  Mission  Area  Template: 

Avionics  Test  Resources  (Also  see  Attachment  3,  T&E  Resources,  for  details) 
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Table  3.1.  Airframe  Modeling  and  Simulation  (M&S). 


Typical  Models 

Purpose 

Ballistic  Accuracy  Models 

Provide  flow  field  effects  coefficients  for  OFP  for 
initial  SEEK  EAGLE  tests 

Control  systems  block  diagrams  (flight  control, 
mechanical  subsystems,  etc.) 

System  stability  analysis 

System  architecture  models  (flight  control, 
mechanical  subsystems,  etc.) 

Inter-item  requirements  determination.  Intra-sys¬ 
tem  (hardware/software)  requirements  determina¬ 
tion,  failure  mode  analysis,  RM&A  analysis 

Structural  Models 

Loads,  strength,  life,  aeroelastic,  weight  and  bal¬ 
ance,  vibroacoustic. 

Aerodynamic  Models 

Digital  and  piloted  simulation,  aircraft  perfor¬ 
mance  analysis,  flight  control  development,  flut¬ 
ter  analysis,  air  data  computation  development, 
determine  forces  and  moments,  inlet  performance 
and  airframe/engine  compatability. 

Specialty  Models  (models  with  very  specific  pur¬ 
pose  used  during  systems  engineering) 

Simulate  items  (components,  subsystems,  sys¬ 
tems)  functionality  during  systems  engineering 
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Typical  Models 

Purpose 

Component/subsystem  vulnerability  models 

Assess  test  item  vulnerability.  Contact  Wright 
Labs/FIV  for  further  information 

In-flight  simulators  and  instrumented  flying  test 
beds 

Variable  in-flight  stability  training  aircraft 
(VISTA),  total  in-flight  simulator  (TIPS)  (both 
Calspan  Co.  operated),  and  contractor  owned/ 
operated  aircraft 

Table  3.2.  Airframe  Measurement  Facilities  (MF). 


Typical  Tests 

Resource 

(these  facilities  are  usually  not  specific  to  test 
item  .i.e.,  independent  of  test  item  size,  shape 
architecture,  etc.) 

Conceptual  design  wind  tunnel  tests  (platform 
shaping,  new  technologies  experiments,  prelimi¬ 
nary  aerodynamics,  etc.) 

Contractor  and  government  (gov’t).  Wind  Tun¬ 
nels  [Arnold  Engineering  Development  Center 
(AEDC),  Wright  Lab,  etc.] 

Gather  detailed  wind  tunnel  data  for  aerodynamic 
databases  (performance  and  flight  control  data, 
and  spin  recovery  data) 

Contractor  and  gov’t.  Wind  Tunnels  (AEDC, 
Wright  Lab,  etc.) 

Wind  tunnel  flutter  tests 

Contractor  and  gov’t.  Wind  Tunnels  (AEDC, 
Wright  Lab,  etc.) 

Structural  strength,  life,  and  stiffness  tests 

Primarily  contractor  owned/operated  test  stands, 
Wright  Labs  Structural  Test  Facility 

Component-level  Environmental  Tests  (strength, 
life,  functionality  and  acceptance  tests) 

Acceptance  testing  is  typically  performed  in  con¬ 
tractor  facilities/chambers. 

Gov’t  Facilities:  Erosion  control  measurement 
facilities,  component  chmatic  chambers  Central 
Inertial  Guidance  Test  Facility  (CIGTF)  at  Hollo¬ 
man  AFB,  large  climatic  chamber  at  Eglin  AFB, 
Environment  Test  Faclities  at  WPAFBHvpersonic 
ranges  at  AEDC 

Birdstrike  Effects 

Birdstrike  Impact  Facilities  at  AEDC  and  Wright 
Labs 

Ballistic  Accuracy  Wind  Tunnel  Tests 

Government  wind  tunnels,  AEDC  Aerodynamic 
Wind  Tunnel  4T  and  Propulsion  Wind  Tunnel 

16T 

Safe  Separation  Weapon  Integration  Tests 

Government  wind  tunnels,  AEDC  Aerodynamic 
Wind  Tunnel  4T  and  Propulsion  Wind  Tunnel 

16T. 

Vectored  Nozzle  Performance  Testing 

AEDC  Propulsion  Wind  Tunnel  16T 
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Table  3.3.  Airframe  System  Integration  Labs  (SIL). 


Typical  Tests 

Resource 

(hardware  is  typically  modeled  in  software  or  is 
prototype  and  the  resources  are  usually  specific  to 
the  test  items) 

Bench-Top  component  lab  tests  (for  example,  actu¬ 
ator  bench  tests,  control  surface  load  test,  software 
modules  tests,  etc.) 

Usually  performed  in  contractor  facilities  or  in  Air 
Logistics  Center  (ALC)  facilities  for  P^I  programs 

Subsystem  lab  tests  (for  example,  software  based 
flight  control/avionics  integration  tests,  piloted  fly¬ 
ing  qualities/flight  controls  simulation,  prototype 
mechanical  subsystems  tests) 

Primarily  performed  in  contractor  facilities  for 

o 

new  system  acquisition  or  in  ALC  facilities  for  P  I 
programs 

Preliminary  logistics  related  tests  (gather  prelimi¬ 
nary  maintenance  task  related  data  using  full-scale 
mockups) 

Mockups  are  usually  built  in  contractor  facilities 
for  new  system  acquisition 

Preliminary  (software  based)  failure  modes  and 
effects  criticality  tests 

Contractor  hosted  computer  models 

Fielded  OFP  Support/Mod 

Applicable  ALC  (type  aircraft  dependent) 

New  Technology  demonstrations 

-  Ground  based:  high  pressure  hydraulics,  virtual 
cockpit,  human  factors/life  support,  etc. 

Wright  Labs  (materials  lab,  flight  control  lab,  etc.). 
Human  Systems  Center  (HSC)  labs  (human  factor 
labs,  centrifuge,  etc.),  and  contractor  labs 

Ballistic  Accuracy  Mods  to  OFP 

Air  Force  SEEK  EAGLE  Office  (AFSEO) 

Table  3.4.  Airframe  Hardware-In-The-Loop  (EQTL)  Facilities. 


Typical  Tests 


Resource 


(these  facilities  test  uninstalled  hardware  and  are 
typically  test  item  specific) 


Flight  control/hydraulic  integration  tests  (iron  bird  I  Usually  performed  in  contractor  facilities  (hard- 


tests) 


Pre-installed  structural  loads  tests 


Gyro/attitude  control  system  tests 


ware  specific) 


Usually  performed  in  contractor  facilities  (hard¬ 
ware  specific) 


Usually  performed  in  contractor  facilities  (hard¬ 
ware  specific) 


HITL  piloted  simulations  (to  assess  system  stabil-  Usually  performed  in  contractor  facilities  (hard¬ 
ily)  ware  specific) 

Mechanical  systems  functionality  tests  (fuel  sys-  Usually  performed  in  contractor  facilities  (hard- 
tem  tilt  tests,  electrical  power  load  tests,  etc.)  ware  specific) 

Component/subsystem  life-cycle  endurance  tests/  Usually  performed  in  contractor  facilities  (hard- 


Qualification  Tests 


ware  specific) 
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Typical  Tests 

Resource 

Failure  Modes  and  Effects  Analysis  Testing 

Usually  performed  in  contractor  facilities  (hard¬ 
ware  specific) 

Fielded  OFF  Support/Mod 

Applicable  ALC 

Table  3.5.  Airframe  Installed  System  Test  Facilities  (ISTF). 


Typical  Tests 

Resource 

(these  facilities  test  items  as  installed  in  the  air 
vehicle) 

Ground  vibration  tests  (GVT)  -  (determine  struc¬ 
tural  mode/freq.) 

Limited  gov’t  facilities  (Edwards  AFB, ),  usually 
conducted  using  contractor  facilities 

Limit  cycle/ground  resonance  test  (GRT)  -  (evalu¬ 
ate  open  and  closed  loop  stability  of  flight  con¬ 
trols,  mechanical  subsystems,  etc.) 

Limited  gov’t  facilities  (Edwards  AFB),  usually 
conducted  using  contractor  facilities 

EMI/EMC  tests 

AFDTC  Pre-flight  Integration  of  Munitions  and 
Electronic  Systems  (PRIMES)  and  Large  shielded 
Benefield  anechoic  facility  (BAF)  at  AFFTC 

Aircraft  ground  load  tests  (drop  tests,  structural 
failure  tests  (100%,  200%  load  tests) 

Usually  conducted  at  Contractor  facilities  (Lock¬ 
heed  facility  at  Plant  42,  Palmdale,  Ca.  and 

Wright  Labs 

Live  Fire  Tests 

Wright  Labs/FIV  (up  to  30mm),  high  speed  test 
track  at  Holloman  AFB 

Full-scale  climatic  tests 

McKinley  Climatic  Lab,  Eglin  AFB 

System  functional/acceptance  tests 

Conducted  at  manufacturing  site 

Prototype  Ejection/Egress  Systems  Tests  (hyper¬ 
sonic  and  crew  module  ejection  systems) 

high  speed  test  track  (Holloman  AFB) 

Cruise  Missile  T&E 

Full  scale  captive  flight  testing  in  AEDC  Propul¬ 
sion  Wind  Tunnel  16T 

Electronic  Countermeasures  (ECM)  and  Elec¬ 
tronic  Counter-Countermeasures  (ECCM)  tests 
(flare  dispensing,  decoys,  chaff,  etc.) 

High  speed  test  track  (Holloman  AFB) 

Ground  Gun  Tests 

Gun  Test  Facility  at  Edwards  AFB  and  Eglin  AFB 

Table  3.6.  Open  Air  Ranges  (OAR)  -  Airframe  related  T&E  (Typical  Ground/Pre-flight  Tests). 


Typical  Ground/Pre-flight  Tests 

Resource 

(see  Attachment  3  for  hst  and  description  of 
major  test  locations/ranges) 

Suitability  Ground  Tests  (Reliability,  Maintain¬ 
ability,  etc.) 

Conducted  at  test  locations 
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Typical  Ground/Pre-flight  Tests 

Resource 

Pre-flight  taxi  tests  (assess  handling  qualities,  sys¬ 
tems  functionality  with  system  in  motion,  etc.) 

Conducted  at  weapon  system  manufacturing  site 

Ground  Operations  Tests  (turn  radius,  handling 
qualities,  external  lighting,  etc.) 

Conducted  at  weapon  system  manufacturing  site 
prior  to  delivery  then  at  test  site  during  gov’t  T&E 

UAV/Cruise  Missile  “IRON  BIRD”  Ground 
Launch  tests 

Utah  Test  and  Training  Range  (UTTR) 

Braking,  RTO  tests,  arresting  systems  tests 

These  are  usually  high  risk  tests,  and  are  typically 
conducted  at  Edwards  AFB  (emergency  options) 

EMI/EMC  test 

PRIMES  Test  Facility  at  Eglin,  B  AF  at  AFFTC 

Table  3.7.  Open  Air  Ranges  (OAR)  -  Airframe  related  T&E  (Typical  Flight  Tests). 


Typical  Flight  Tests 

Resources 

Envelope  expansion 

-  Flight  controls,  flutter,  mechanical  systems,  80/ 
100%  loads,  flying  qualities,  air  data,  high  AO  A, 
etc. 

-  New  aircraft/major  P^I/high  risk  programs  - 
Edwards  AFB,  Eglin  AFB 

Aero/Performance  Data  Base  Verification 

Edwards  AFB 

Mission  Systems: 

-  Air  refueling  qualification 

-  In-flight  icing  and  rain 

-  Aerial  cargo  delivery  development 

and  OT&E 

-  Initial  weapon  separation  &  delivery  (not  certi¬ 
fication  or  weapon  development) 

-  Terrain  following/terrain  avoidance  (TF/TA) 
development  (flyup  command  checkout,  ride 
quality,  auto  let  down,  etc.  Also  see  Avionics 
templates  -TF/TA) 

Instrumented  KC-135  and  KC-10  aircraft  -412 
TW,  Edwards  AFB 

Instrumented  KC-135aircraft  -412  TW,  Edwards 
AFB 

Edwards  AFB, ,  Eglin  AFB 

Edwards  AFB,  Eglin  AFB,  UTTR 

Edwards  AFB 

UAV/Cruise  Missile  captive  carriage  testing 

UTTR,  Edwards  AFB 

UAV  testing 

UTTR,  Eglin,  White  Sands  Missile  Range 
(WSMR) 

Cruise  Missile  T&E 

UTTR,  Eglin,  Vandenburg  Test  Range 

Fielded  OFP  Support/Mod 

Applicable  OFP 

74 


AFMAN99-110  3  JULY  1995 


Figure  3.14.  Propulsion  Mission  Area. 
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Table  3.8.  Propulsion  Modeling  and  Simulation  (M&S). 


Typical  Models 

Purpose 

Performance  Deck:  steady-state  engine  and  com¬ 
ponents 

Engine,  component,  and  Aircraft  performance 
analysis,  first  order  usage  analysis 

Specification  Deck:  steady  state  engine  &  compo¬ 
nents  as  baselined  in  the  specification 

Acceptance  testing.  Technical  Order  (TO)  prepa¬ 
ration  analysis 

Transient  Deck:  transient  engine/component  per¬ 
formance 

Engine  component,  control  system,  ancillary  sys¬ 
tems,  thermo,  maintenance,  and  structural  perfor¬ 
mance;  second  order  usage  analysis 

Aerodynamic  Models  (May  interface  with  air¬ 
frame  aero  model) 

Assess  thermal  and  structural  effects  and  help  per¬ 
form  durability  predictions 

Structural  Models  (may  interface  with  airframe 
structural  models) 

loads,  deflections,  stress/strain  analysis  and  dura¬ 
bility  predictions 

Inlet  Performance  Model 

Engine  face  distortion  (static  and  dynamic  pres¬ 
sure) 

Scram  Jet  Performance 

Combustion  chemistry  and  internal  flow  Model¬ 
ing 

External  burning  afterbody 

Surface  conditions  and  thrust 
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Table  3.9.  Propulsion  Measurement  Facilities  (MF). 


Typical  Tests 

Resource 

(these  facilities  are  usually  not  specific  to  test 
item  i.e.,  independent  of  test  item  size,  shape 
architecture,  etc.) 

Material  properties  development  and  T&E 

Material  vendor/subcontractor  facilities,  engine 
manufacturer  and  gov’t  material  Lab  (Wright 

Lab  for  example) 

Airfoil  cascade  performance 

Engine  manufacturer  facilities,  gov’t  labs  and 
cascade  rigs  (Wright  Lab  for  example). 

Component  performance  and  durability  tests 

Usually  contractor  but  some  gov’t  rigs  (Wright 
Labs,  AEDC,  and  NASA-Lewis)  to  test:  bear¬ 
ings,  cascades,  combustors,  pumps,  and  seals. 

Component  performance,  stability,  and  struc¬ 
tural  integrity 

Usually  contractor,  but  some  gov’t  component 
rigs  for  fans,  compressors,  and  nozzles.  (Wright 
Labs  and  AEDC). 

Inlet  distortion  testing  using  Air  Jet  Distortion 
Generator  or  scale  models 

AEDC  engine  test  cells  or  wind  tunnels 

Engine  inlet  icing,  deicing 

AEDC  engine  test  cells 

Supersonic  flow  inlets 

AEDC  Aerospace  Propulsion  Systems  Test 
Facility  (ASTF),  Tunnels  B,  C,  and  APTU 

Transonic  Performance  of  Scramjet  Engine 

AEDC  ASTF 

Flight  Test  Engine  Calibration 

AEDC  altitude  cells 

Table  3.10.  Propulsion  System  Integration  Labs  (SIL). 


Typical  Tests 

Resource 

(hardware  is  typically  modeled  in  software  or  is 
prototype  and  the  resources  are  usually  specific 
to  the  test  items) 

Bench-Top  tests  for  pumps,  actuators,  heat 
exchangers,  and  engine  controls 

Usually  performed  in  contractor  facilities  or  in 
ALC  facilities  for  P^I  programs 

Aero-propulsion  labs  for  developing  lead¬ 
ing-edge  technologies 

Aero-Propulsion  Directorate,  Wright  Labs 

Preliminary  (software  based)  failure  modes  and 
effects  criticality  tests 

Contractor  hosted,  with  interfaces  computer 
modeled 
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Table  3.11.  Propulsion  Hardware-In-The-Loop  (HITL)  Facilities. 


Typical  Tests 

Resource 

(these  facilities  test  uninstalled  hardware  and  are 
typically  test  item  specific) 

Integrated  control  system  tests  (with  flight  con¬ 
trols/avionics),  engine  control  tests,  sensor  and 
effectors  T&E 

Usually  performed  in  contractor  facilities  (hard¬ 
ware  specific). 

Uninstalled  engine  sea-level  testing:  perfor¬ 
mance,  operability,  durability,  and  failure  mode/ 
effects  (vulnerability/containment  tests) 

Usually  performed  in  contractor  facilities  (hard¬ 
ware  specific).  Gov’t  facilities  include  AEDC. 

Uninstalled  engine  altitude  testing:  performance, 
operability,  stability,  inlet/nozzle  compatibility 

AEDC.  Limited  engine  contractor  facilities 

Uninstalled  engine  anti-icing/de-icing  qualifica¬ 
tion 

AEDC 

Altitude  chamber  testing  performance  and  opera¬ 
bility  of  full-up  inlet/engine  combination  at  vari¬ 
eties  of  sideslips  and  angles-of-attack 

AEDC-Aerospace  Propulsion  Systems  Test  Facil¬ 
ity  (ASTF) 

Sea  level  tests  of  uninstcilled  engines  at  extreme 
temperatures 

McKinley  Labs  (Eglin  AFB)  engine  test  cells 

Uninstalled  test  at  extreme  pressures  and  temper¬ 
atures 

AEDC,  and  NASA-Lewis 

Table  3.12.  Propulsion  Installed  System  Test  Facilities  (ISTF). 


Typical  Tests 

Resources 

(these  facilities  test  items  as  installed  in  the  air 
vehicle) 

Installed  engine  tests  (ground  level,  ambient  con¬ 
ditions)  of  steady-state  and  transient  performance 
thrust  characteristics 

Horizontal  Thrust  Stand,  Edwards  AFB 

Installed  engine  ground  tests  at  climatic  extremes 

McKinley  Climatic  Lab  (Eglin  AFB) 

Live  Fire  Vulnerability  Tests  with  operating 
engine  (stationary  and/or  with  blown  air) 

Wright  Labs/FIV. 

Accelerated  mission  testing;  emission  testing; 
fuel  evaluation  testing;  endurance  testing;  and 
component  testing  for  future  sea-level  standard 
engine  test  requirements 

OC-ALC 

System  functional/acceptance  tests 

Performed  at  aircraft/engine  mating  location 
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Table  3.13.  Open  Air  Ranges  (OAR)  -  Propulsion  T&E  (Typical  Ground/Pre-flight  Tests). 


Typical  Ground/Pre-flight  Tests 

Resource 

(see  Attachment  3  for  list  and  description  of 
major  test  locations/ranges) 

Suitability  Ground  Tests  (Reliability,  Maintain¬ 
ability,  etc.) 

Conducted  at  test  locations  -  Major  Range  and 
Test  Facility  Bases  (MRTFB),  ALCs,  and  other 
Department  of  Defense  (DOD)  &  contractor 
ranges 

Ground  Engine  starts  in  various  wind  conditions 
and  ambient  temperatures 

Conducted  at  test  locations  -  MRTFBs,  ALCs, 
and  other  DOD  &  contractor  ranges 

Pre-flight/functional  checks:  steady-state,  tran¬ 
sient,  and  back-up  control  operation 

Conducted  at  test  locations  -  MRTFBs,  ALCs, 
and  other  DOD  &  contractor  ranges 

Ground  Acoustics  (danger  areas,  noise  levels, 
etc.)  and  engine  exhaust  blast  zones 

Conducted  at  test  locations  -  MRTFBs,  ALCs, 
and  other  DOD  &  contractor  ranges  if  appropri¬ 
ate  equip,  available. 

Thrust  reverser  operations  in  various  X-winds. 
Ground  operations  (ops)  using  thrust  reverse 

Conducted  at  test  locations  -  MRTFBs,  ALCs, 
and  other  DOD  &  contractor  ranges 

Table  3.14.  Open  Air  Ranges  (OAR)  -  Propulsion  T&E  (Typical  Flight  Tests). 


Typical  Flight  Tests 

Resources 

Envelope  Expansion 

-  steady-state  and  transient  engine  ops  through¬ 
out  flight  envelope,  afterburner  operation, 
back-up  control  testing,  in-flight  thrust  reversing, 
airstarts,  thrust  vectoring.  High  AOA,  and 
departed  flight. 

Conducted  at  test  locations  -  MRTFBs,  ALCs, 
and  other  DOD  ranges.  Typically  next  genera¬ 
tion  engines  in  single  engine  aircraft  and  high 
AOA  testing  is  conducted  at  the  AFFTC  (emer¬ 
gency  landing  options  on  lakebeds). 

Propulsion  Performance 

-  aircraft  accelerations,  climbs,  descents,  cruise, 
and  turns 

Conducted  at  test  locations  -  MRTFBs,  ALCs, 
and  other  DOD  ranges. 

Engine  armament  gases  compatibility  (gun  gas 
ingestion,  missile  firing,  etc.) 

Conducted  at  test  locations  -  MRTFBs,  ALCs, 
and  other  DOD  ranges. 

Alternate  fuels  T&E  (Jp-5,  JetA,  etc.) 

Conducted  at  test  locations  -  MRTFBs,  ALCs, 
and  other  DOD  ranges. 
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Figure  3.15.  Avionics  Mission  Area. 


Table  3.15.  Avionics  Modeling  and  Simulation  (M&S). 


Typical  Models 

Purpose 

Mission  Level  Survivability/Threat  Models 

Mission  level  analysis  to  support  COEAs, 

ORDs,  and  Electronic  Combat  (EC)  system  per¬ 
formance.  These  models  are  usually  VV&A’d  by 
National  Air  Intelligence  Center  (NAIC)/TANV, 
Wright-Patterson  AFB 

Indoor  EC  threat  simulators 

Assess  the  effectiveness  of  EC  systems/integra¬ 
tion  in  an  indoor,  controlled  environment.  Typi¬ 
cal  simulators  include  Real-time  Electronic 
Digitally  Controlled  Analyzer  Processor  (RED¬ 
CAP)-  46th  TW,  Buffalo  NY.  and  various  Wright 
Labs/AAWA. 
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Typical  Models 

Purpose 

Avionics  Models 

Digital  system  models  or  digital  system  descrip¬ 
tion  of  avionics  on  selected  aircraft.  The  IFAST 
facility  has  avionics  models  of  selected  systems 
on  B-IB,  F-15,  F-16.  When  hardware  is  not 
available,  then  models  are  substituted  to  provide 
system  integration  test  capability. 

Navigation  Models 

Simulate  nav.  system  inputs  to  assess  perfor¬ 
mance.  These  models  are  typically  contractor 
developed,  but  Central  Inertial  Guidance  Test 
Facility  (CIGTF),  Holloman  AFB  may  also  pro¬ 
vide  assistance. 

Specialty  Models  (models  with  very  specific  pur¬ 
pose  used  during  systems  engineering) 

Simulate  items  (sensor,  CNI,  components,  sub¬ 
systems,  systems)  functionality  during  systems 
engineering 

Component/subsystem  vulnerability  models 

Assess  test  item  vulnerability.  Contact  Wright 
Labs/FIV  for  further  information 

In-flight  simulators  and  instrumented  flying  test 
beds 

Evaluate  avionic  subsystem  and  component  per¬ 
formance  in  instrumented  flying  test  beds  in  Fly¬ 
ing  Test  Beds,  section  of  Attachment  2,  VII.C. 
Example  includes  Advanced  Radar  Test  Bed. 
Also  see  Support  Aircraft  Test  Capability  Master 
Plan  (TCMP). 

Table  3.16.  Avionics  Measurement  Facilities  (MF). 


Typical  Tests 

Resource 

(these  facilities  are  usually  not  specific  to  test 
item  i.e.,  independent  of  test  item  size,  shape 
architecture,  etc.) 

Sensor  characterization  facilities  (antenna  pat¬ 
tern,  EO  test  facility,  etc.) 

Various  Rome  Lab  Facilities  (Griffiss  AFB,  NY) 
and  contractor  facilities. 

Signature  measurement 

Contractor  and  gov’t,  ground  ranges.  Examples 
include  Radar  Target  Scatter  Facility 
(RATSCAT)/RATSCAT  Advanced  Measurement 
Facility  (RAMS)  (Holloman  AFB)  and  infrared 
(IR)  measurement  labs 

In-flight  measurement  (Radar,  IR,  acoustic,  etc.): 
see  special  access  required  (SAR)  facilities  in 
Attachment  3  for  POC  #s. 

Guidance  and  Navigation  system  characteriza¬ 
tion  and  certification 

CIGTF  -  46TG/rGG,  Holloman  AFB.(Guidance 
and  navigation  component  and  system  laboratory 
and  field  tests). 
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Table  3.17.  Avionics  System  Integration  Labs  (SIL). 


Typical  Tests 

Resource 

(hardware  is  typically  modeled  in  software  or  is 
prototype  and  the  resources  are  usually  specific  to 
the  test  items) 

Bench-Top  component  lab  tests  (for  example, 
evaluate  gyro  accuracy,  assess  sensor  resolution, 
etc.).  These  are  end-to-end  tests  of  components. 

Usually  performed  in  contractor  facilities  or  in 

ALC  facilities  for  P^I  programs,  but  gov’t  exam¬ 
ples  include  Electronic  Warfare  Avionics  Integra¬ 
tion  Support  Facility  (EWAISF)  (WR-ALC/LN), 
IFAST  (412  TW,  Edwards  AFB),  and  CIGTF 
(Holloman  AFB) 

Subsystem  level  lab  tests.  These  tests  evaluate 
intra  and  inter  subsystem  integration  using  prima¬ 
rily  computer  simulation  of  inputs/outputs. 
Examples  include  evaluating  display  interface 
with  radar  system,  ensuring  EC  subsystem  inter¬ 
faces  with  avionics,  and  software  module  tests 

Primarily  performed  in  contractor  facilities  for 
new  system  acquisition  or  in  ALC  facilities  for 

P^I  programs.  Gov't  examples  include  EWAISF 
(WR-ALC/LN),  IFAST  (412  TW,  Edwards  AFB), 
CIGTF  (Holloman  AFB),  and  PRIMES  (Eglin 
AFB). 

Preliminary  logistics  related  tests  (gather  prelimi¬ 
nary  maintenance  task  related  data  using 
full-scale  mockups) 

Mockups  are  usually  built  in  contractor  facilities 
for  new  system  acquisition 

Preliminary  (software  based)  failure  modes  and 
effects  criticality  tests 

Contractor  hosted  computer  models 

Fielded  OFP  Support/Mod 

Applicable  ALC 

New  Technology  demonstrations/research  (ring 
laser  advancement,  GPS  enhancement,  radar 
advancements,  etc.) 

ASC/Wright  Labs  (avionics  directorate),  HSC 
labs  (human  factor  labs,  and  contractor  labs 

Table  3.18.  Avionics  Hardware-In-The-Loop  (HITL)  Facilities. 


Typical  Tests 

Resource 

(these  facilities  test  uninstalled  hardware  and  are 
typically  test  item  specific) 

Assess  Avionics  Interface  with  non-avionics  sub¬ 
systems  (similar  to  SIL  tests  but  using  actual 
hardware).  Examples  are  avionics-to  weapon 
delivery  interface,  avionics  to  flight  controls,  avi¬ 
onics  to  BIT,  and  EC  integration. 

Usually  performed  in  contractor  facilities  (hard¬ 
ware  specific).  Gov't  example  is  the  Guided 
Weapon  Evaluation  Facility  (GWEF)-Eglin 

AFB),  REDCAP  (Calspanj^uffalo),  Air  Force 
Electronic  Warfare  Evaluation  Simulator 
(AFEWES)  (46TW/Fort  Worth)  and  IFAST 
(412TW/Edwards  AFB)  and  PRIMES  Test  Facil¬ 
ity  (Eglin  AFB) 

Navigation  subsystem  tests 

Usually  performed  in  contractor  facilities  (hard¬ 
ware  specific).  Gov't  facility  is  CIGTF  at  Hollo¬ 
man  AFB 
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Typical  Tests 

Resource 

Sensor  functional  tests 

Usually  performed  in  contractor  facilities  (hard¬ 
ware  specific) 

Cockpit  systems  end-to-end  tests  (helmet 
mounted  displays,  night  vision  system  test,  mis¬ 
sion  management  system  tests. 

Usually  performed  in  contractor  faciUties  (hard¬ 
ware  specific) 

Component/subsystem  life-cycle  endurance  tests/ 
Qualification  Tests 

Usually  performed  in  contractor  facilities  (hard¬ 
ware  specific) 

Failure  Modes  and  Effects  Analysis  Testing 

Usually  performed  in  contractor  facilities  (hard¬ 
ware  specific) 

Fielded  OFF  Support/Mod 

Applicable  ALC 

Table  3.19.  Avionics  Installed  System  Test  Facilities  (ISTF). 


Typical  Tests 

Resource 

(these  facilities  test  items  as  installed  in  the  air 
vehicle) 

Onboard  Offensive/Defensive  suite  evaluation, 
antenna  radiation  coverage  measurements,  sys¬ 
tems  sensitivity  measurements. 

Benefield  Anechoic  Facility  (BAF-  Edw  ards 
AFB)  PRIMES  Test  Facility  (EglinAFB) 

Onboard  simulation  (simulates  air-to-air  (A/ A) 
engagements,  air-to-ground  (A/G)  missions,  etc., 
through  onboard  displays  [heads-up  display 
(HUD),  cathode  ray  tubes  (CRT),  etc.) 

Typically  performed  at  RTO  facility/program  test 
location.  PRIMES  Test  Facility  (Eglin  AFB) 

EMI/EMC  tests 

Limited  test  capability  at  some  test  centers  [BAF, 
Pre-flight  Integration  of  Munitions  and  Elec¬ 
tronic  Systems  (PRIMES)]  and  contractor  facili¬ 
ties. 

System  functional/acceptance  tests 

Performed  at  manufacturing  plant 

Live  Fire  Tests 

Wright  Labs/FIV  (up  to  SOmm),  &  rocket  sled 
test  track  at  Holloman  AFB 

Full-scale  climatic  tests 

McKinley  Climatic  Lab,  Eglin  AFB 

Table  3.20.  Open  Air  Ranges  (OAR)  -  Avionics  T&E  (Typical  Ground/Pre-flight  Tests). 


Typical  Ground/Pre-flight  Tests 

Resource 

(see  Attachment  3  for  list  and  description  of 
major  test  locations/ranges) 

Suitability  Ground  Tests  (ReUability,  Maintain- 
abihty,  etc.) 

Conducted  at  test  locations  -  MRTFBs,  ALCs, 
and  other  DOD  ranges 

Weapon  system  harmonization  tests  (gun  sys¬ 
tem/sensor  boresighting,  HUD/  windscreen 
parallax/light  transmissivity  tests,  etc.) 

Conducted  at  test  location  utilizing  local 
resources  (gun  butt,  theodolites,  etc.) 
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Typical  Ground/Pre-flight  Tests 

Resource 

Weapon/stores  drop  tests  (into  sand  box  or  sim¬ 
ilar  fixture) 

Conducted  at  test  location  -  MRTFBs,  ALCs, 
and  other  DOD  ranges 

BIT  pre-flight 

Conducted  at  test  location  -  MRTFBs,  ALCs, 
and  other  DOD  ranges 
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Table  3.21.  Open  Air  Ranges  (OAR)  -  Avionics  T&E  (Typical  Flight  Tests). 


Typical  Flight  Tests 

Resources 

Sensor  development  (test  radar  modes,  EO/IR 
performance,  etc.) 

AFFTC,  AFDTC,  Utah  Test  and  Training  Range 
(UTTR),  and  ALCs  (and  their  local  area  ranges  - 
for  example,  including  China  Lake  for  AFFTC 
airspace) 

Navigation  /Guidance  development  (box  routes, 
radar/star  tracker/etc.,  integration  with  nav.  sub¬ 
system) 

CIGTF  46  TG/TGG  (Holloman  AFB,  NM). 

Box  Routes  may  vary  from  small  race-track  type 
patterns  within  DOD  airspace  or  could  also 
include  routes  across  thousands  of  miles. 

Comm  and  Ident  T&E  (max.  functional  range, 
azimuth  characterization,  etc.) 

Conducted  at  test  locations  (if  part  of  larger 
T&E)  and  MRTFBs. 

Antenna  pattern  tests 

Owens  Valley  Range 

Mission/Support  Systems  development  (night 
vision,  GCAS,  BIT,  HMD,  onboard  simulation, 
etc.) 

AFFTC 

In-flight  signature  measurement 

See  MFs  and  SAR  resources  (see 

Attachment  2,  Avionics  OARs) 

Avionics/EC  suite  integration  T&E 

Areas  with  surrogate,  simulated,  or  actual  threat 
assets  -  Examples  are:  AFFTC,  AFDTC,  and 
SAR  resources 

TF/TA  development  (radar/flight  control  inte¬ 
gration,  ground  plane  clearance,  mission  OFP 
integration.  Also  see  Airframe  templates-TF/ 
TA) 

Ranges  with  various  terrain,  topography,  ete. 
AFFTC  usually  performs  early  development 
using  instrumented  routes  (see  Airframe 
resources  in  Attachment  3) 

Weapon  System  Integration  T&E  (weapon  sepa¬ 
ration  T&E,  nav.  integration  w/smart  weapons, 
sensor/guided  weapon  integration,  avionics/ 
weapon  integration,  etc.,  gun  harmonization) 

AFFTC,  UTTR,  and  Air  Force  Development 

Test  Center  (AFDTC)  ranges  (also  see  Airframe 
resources  in  Attachment  3). 

Preliminary  Survivability  T&E  [Minimum 
(Min)Resolvable  Temp.(MRT),  detection,  etc.] 

AFFTC,  AFDTC,  and  SAR  resources 

Interoperability  T&E 

JITC  (also  see  interoperability  section  of  Mis¬ 
sion  Reqm’ts  template) 

Avionics  OT&E  (survivability,  SEAD,  weapon 
delivery,  etc.). 

AFFTC,  AFDTC,  and  SAR  resources 

Fielded  OFP  Support/Mod 

Applicable  ALC,  AFFTC,  and  AFDTC 

JOSEPH  W.  RALSTON,  Lt  General,  US AF 
DCS,  Plans  and  Operations 
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AIS — ^Automated  Information  System 

AISF — ^Avionics  Integration  Support  Facility 

ALC — ^Air  Logistics  Center 

AMT — ^Accelerated  Mission  Testing 

AOA — ^Angle-of-Arrival 

AOA — ^Angle-of-Attack 

A-P-A — Airframe-Propulsion-Avionics 

APU — Auxiliary  Power  Unit 

ARTB — ^Advanced  Radar  Test  Bed 

ASC — ^Aeronautical  Systems  Center 

ASETS — Airborne  Seeker  Evaluation  Test  System 

ASTF — ^Aeropropulsion  Systems  Test  Facility 

ATRIS — Automated  Test  Resources  Information  System 

ATIC — ^Avionics  Test  and  Integration  Complex 

BAF — ^Benefield  Anechoic  Chamber 

BIT — Built-In-Test 

BLM — ^Bureau  of  Land  Management 

c.g — Center  of  Gravity 

C4I — Command,  Control,  Communications,  Computers  and  Intelligence 

CAIS — Common  Airborne  Instrumentation  System 
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CDR — Critical  Design  Review 

CDRL — Contract  Data  Requirements  List 
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CNI — Communications,  Navigation,  and  Identification 
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COI — Critical  Operational  Issue 
CONORS — Concept  of  Operations 
COTS — Commercial-Off-The-Shelf 
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CRT — Cathode  Ray  Tube 

CRWG — Computer  Resources  Working  Group 

CSC — Computer  Software  Component 

CSCI — Computer  Software  Configuration  Item 

CSEL — Communications  Systems  Evaluation  Lab 

CSU — Computer  Software  Unit 

CTF — Combined  Test  Force 

DAO  A — ^Dynamic  Angle  Of  Arrival 

DAP — Data  Analysis  Plan 

DEES/CEESIM— Dynamic/Combat  Electromagnetic  Environment  Simulator 

DemA^al — DemonstrationWalidation 

DIME — ^Dynamic  Infrared  Missile  Evaluator 

DISA — Defense  Information  Systems  Agency 

DOD — ^Department  Of  Defense 

DODD — ^Department  Of  Defense  Directive 

DODI — Department  Of  Defense  Instruction 

DODM — Department  Of  Defense  Manual 

DR — Deficiency  Report  (replaced  Service  Report) 

DSMC — ^Defense  Systems  Management  College 
DT&E — ^Developmental  Test  and  Evaluation 
DTIC — Defense  Technical  Information  Center 
EC — Electronic  Combat 
ECCM — ^Electronic  Counter-Countermeasures 
ECIT — ^Electronic  Combat  Integrated  Test 
ECM — ^Electronic  Countermeasures 
ECS — ^Environmental  Control  System 

ECSRL — ^Electronic  Combat  Simulation  Research  Laboratory 

EISF — ^Extendible  Integration  Support  Facility 

EM — ^Electromagnetic 

EMC — ^Electromagnetic  Compatibility 

EMD — ^Engineering,  Manufacturing,  and  Development 

EMI — ^Electromagnetic  Interference 
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EO — ^Electro-Optical 

EOA — Early  Operational  Assessment 

EOSEM — ^Electro-Optical  Simulation 

ESC — ^Electronic  Systems  Center 

ESS — ^Electromagnetic  System  Simulator 

ETF — ^Engine  Test  Facility 

EW — ^Electronic  Warfare 

EWAISF — ^Electronic  Warfare  Avionics  Integration  Support  Facility 

FAA — ^Federal  Aviation  Administration 

FICSIM — ^Fire  Control  Simulation 

FLIR — Forward  Looking  Infrared 

FMEA — ^Failure  Modes  and  Effects  Analysis 

FOT&E — ^Follow-on  Operational  Test  and  Evaluation 

FOV — ^Field-of-View 

GCAS — Ground  Collision  Avoidance  System 

GENESIS — GENeralized  Environment  for  the  Simulation  of  Integrated  Systems 

GFE — Government  Furnished  Equipment 

GHz — Giga  Hertz 

Gov't — Government 

GPS — Global  Positioning  System 

GRT — Ground  Resonance  Test 

GVT — Ground  Vibration  Test 

GWEF — Guided  Weapons  Evaluation  Facility 

HCF — ^High  Cycle  Fatigue 

HF — ^High  Frequency 

HITL — Hardware-In-The-Loop 

HMD — Helmet  Mounted  Display 

HSC — ^Human  Systems  Center 

HUD — Heads-Up  Display 

I&GTC — ^Infrastructure  and  Generic  Test  Capability 
IADS — Integrated  Air  Defense  Systems 
lAL — Integrated  Avionics  Laboratory 
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lAW — In  Accordance  With 
ICD — Interface  Control  Document 

ICNIA — Integrated  Communications,  Navigation,  and  Identification  Avionics 
IDAL — Integrated  Defensive  Avionics  Lab 
EDD — Interface  Design  Document 
Ident — Identification 

lESS — Integrated  Electromagnetic  System  Simulator 

IFAST — ^Integration  Facility  for  Avionic  Systems  Testing 

ILS — Instrument  Landing  System 

ILS — Integrated  Logistics  Support 

ILSP — ^Integrated  Logistics  Support  Plan 

IMP — Integrated  Master  Plan 

IMS — Integrated  Master  Schedule 

INS — Inertial  Navigation  System 

lOT&E — Initial  Operational  Test  and  Evaluation 

IPD — Integrated  Product  Development 

IPX — Integrated  Product  Team 

IR — Infrared 

IRCM — Infrared  Counter  Measures 

IRMP — Integrated  Risk  Management  Process 

ISTF — Installed  Systems  Test  Facility 

ITB — ^Integrated  Test  Bed 

IV&V — ^Installation  Verification  and  Validation 

IWSM — ^Integrated  Weapon  System  Management 

J-STARS — Joint  Surveillance  Target  Attack  Radar  System 

JCS/SI— Joint  Chiefs  of  Staff/Special  Instruction 

JEIO — ^Joint  Effectiveness  and  Interoperability  Office 

JFS — Jet  Fuel  Starter 

JITC — Joint  Interoperability  Test  Center 

JMASS — Joint  Modeling  and  Simulation  System 

JRMET — Joint  Reliability  and  Maintainability  Evaluation  Team 

JTCG/AS — ^Joint  Technical  Coordinating  Group  on  Aircraft  Survivability 
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JTIDS — Joint  Tactical  Information  Distribution  System 
KVA— BCilovoltAmp 

LAPES — ^Low  Altitude  Parachute  Extraction  System 
lbs — ^Pounds 

LCF — ^Low  Cycle  Fatigue 

LCL — ^Laser  Communications  Laboratory 

LFT — ^Live  Fire  Test 

LFT&E^ — ^Live  Fire  Test  and  Evaluation 

LID — ^Laser  IRCM  Development 

LOG  TEST — Logisitics  Test 

LORAN — ^Long  Range  Navigation 

LRIP — Low-Rate  Initial  Production 

LRU — ^Line  Replaceable  Unit 

LSA — ^Logistics  Support  Analysis 

LWT — ^Two  Foot  Water  Tunnel 

M&S — ^Modeling  and  Simulation 

MF — ^Measurement  Facility 

MIL — ^Military 

Min — Minimum 

MIP — ^Material  Improvement  Program 

MMW — Millimeter  Wave 

MNS — ^Mission  Needs  Statement 

MOA — Memorandum  of  Agreement 

Mod — ^Modification 

MOE — Measure  of  Effectiveness 

MOP — ^Measure  of  Performance 

MRT — Minimum  Resolvable  Temp 

MRTFB — ^Major  Range  and  Test  Facility  Base 

MS — ^Milestone 

NAIC — National  Air  Intelligence  Center 
NASTRAN — NASA  Structural  Analysis 
nav — ^Navigation 
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NEPA — ^National  Environmental  Protection  Act 

NLT — No  Later  Than 

O&S — Operation  and  Sustainment 

OA — Operational  Assessment 

OAR — Open  Air  Range 

OFP — Operational  Flight  Program 

OPR — Office  of  Primary  Responsibility 

ops — Operations 

ORD — Operational  Requirements  Document 

OSD — Office  of  Secretary  of  Defense 

OT&E — Operational  Test  and  Evaluation 

OTA — Operational  Test  Agency 

OTP — Operational  Test  Program 

OTS— Off-The-Shelf 

P3I — Preplanned  Product  Improvement 

PAES — ^Performance  Assessment  and  Evaluation  System 

PAMS — ^Precision  Antenna  Measurement  System 

PDR — ^Preliminary  Design  Review 

PI — Program  Introduction 

PID — Program  Introduction  Document 

PIRA — ^Precision  Impact  Range  Area 

PMD — ^Program  Management  Directive 

POC — Point  of  Contact 

PRIMES — Pre-flight  Integration  of  Munitions  and  Electronic  Systems  (Facility) 
pro — ^Participating  Test  Organization 
PWT — Propulsion  Wind  Tunnel 

QOT&E — Qualification  Operational  Test  and  Evaluation 

QT&E — Qualification  Test  and  Evaluation 

RAMS — ^RATSCAT  Advanced  Measurement  Facility 

RASPL — Radar  Analysis  and  Signal  Processing  Laboratory 

RATSCAT — ^Radar  Target  Scatter  Facility 

RCM — ^Requirements  Correlation  Matrix 
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RCS — Radar  Cross  Section 

REDCAP — Real-time  Electronic  Digitally  Controlled  Analyzer  Processor 
RF — ^Radio  Frequency 

RM&A — Reliability,  Maintainability,  and  Availability 

RMCC — Ridley  Mission  Control  Center 

RTF — ^Radar  Test  Facility 

RTG — ^Radar  Target  Generator 

RTO — ^Responsible  Test  Organization 

RTS — Radar  Test  Station 

SAR — Special  Access  Required 

SAR — Synthetic  Aperture  Radar 

SARL — Subsonic  Aerodynamic  Research  Laboratory 

SATCOM — Satellite  Communication 

SCLP — Surrogate  Carrier  Launch  Platform 

SDD — System  Design  Document 

SE— SEEK  EAGLE 

SEAD — Suppression  of  Enemy  Air  Defenses 

SEDS — System  Effectiveness  Data  System 

SEMSS — SE  Management  Support  System 

SER — SEEK  EAGLE  Request 

SFTC — Single-Face-To-Customer 

SIL — System  Integration  Laboratory 

Sim — Simulation 

SM — Single  Manager 

SMC — Space  and  Missile  Center 

SMM — System  Maturity  Matrix 

SMS — Stores  Management  System 

SOC — Statement  of  Capability 

Spec — Specification 

SPO — System  Program  Office 

SR — Service  Report  (replaced  by  DR) 

SRD — System  Requirements  Document 
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SRR — System  Requirements  Review 
SRS — System  Requirements  Specification 
SSDD — System/Segment  Design  Document 
SSE — Simulation  Support  Environment 
SSS — System/Segment  Specification 
STAR — System  Threat  Assessment  Report 
STD— Standard 
STP — Software  Test  Plan 

SURVIAC — SurvivabilityA^ulnerability  Information  Analysis  Center 

SUT — System  Under  Test 

SWIS — StoresAVeight/and  Inertia  System 

Sys — System 

T&E — Test  and  Evaluation 

TACAN — ^Tactical  Communications  and  Navigation 

TCMP — Test  Capability  Master  Plan 

TECNET — ^Test  and  Evaluation  Computer  Network 

TEMP — ^Test  and  Evaluation  Master  Plan 

TEMS — Test  and  Evaluation  Mission  Simulator 

TERCOM — ^Terrain  Contour  Matching 

TF — ^Terrain  Following 

TF/TA — Terrain  Following/Terrain  Avoidance 

TGF — Trisonic  Gasdynamics  Facility 

TIFS — Total  In-flight  Simulator 

TIMP — Test  Investment  Master  Plan 

TISP — ^Test  Investment  Strategic  Plan 

TM — ^Telemetry 

TPM — Technical  Performance  Measurement 
TO — ^Technical  Order 

TOV&V — ^Technical  Order  Validation  and  Verification 

TPWG — ^Test  Plan  Working  Group 

TR — ^Technical  Report 

TRMP — Test  Resource  Master  Plan 
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TSPI — ^Time  Space  Positioning  Information 

TW — ^Test  Wing 

U.S.C. — United  States  Code 

UAV — ^Unmanned  Aerial  Vehicle 

UHF — ^Ultra  High  Frequency 

USAF — ^United  States  Air  Force 

UTTR — Utah  Test  and  Training  Range 

UV — ^Ultraviolet 

VHF — ^Very  High  Frequency 

VISTA — Variable  In-flight  Stability  Training  Aircraft 

VV&A — Verification,  Validation,  and  Accreditation 

VWT— Vertical  Wind  Tunnel 

WBS — ^Work  Breakdown  Structure 

WSMR — ^White  Sands  Missile  Range 

Terms 

Refer  to  AFI  99-103  Air  Force  Test  Process,  as  well  as  AFM  11-1,  Air  Force  Glossary  of  Standardized 
Terms  and  DOD  Glossary  of  Defense  Acronyms  and  Terms  for  definitions  of  common  terms  used  is  this 
manual. 
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Attachment  2 

SYSTEM  DEVELOPMENT 

A  complete  and  detailed  discussion  of  test  and  evaluation  (T&E)  and  systems  development  is  contained  in 
Test  and  Evaluation  Management  Guide,  Defense  Technical  Information  Center  (DTIC).  This  attachment 
tailors  systems  T&E/development  discussions  as  they  pertain  to  the  Airframe-Propulsion-Avionics 
(A-P-A)  T&E  process. 

A2.1.  Concept  Phase.  During  the  system’s  conceptual  design  phase,  the  Mission  Needs  Statement 
(MNS)  and  Operational  Requirements  Document  (ORD)  (typically  under  development  at  this  stage)  lay  a 
design  foundation  upon  which  the  general  aerodynamic  shape,  size,  weight,  and  system  architecture  of  the 
air  vehicle  can  be  tailored.  Most  design  tradeoffs  within  and  between  functional  categories/disciplines 
occur  at  this  early  stage  of  a  system’s  development  (Acquisition  Phase  0).  Examples  of  tradeoffs  include: 
inlet  location,  platform  shape  [aerodynamics  vs.  radar  cross  section  (RCS)],  internal  vs.  external  weapons 
carriage,  and  wide  dispersion  of  components/many  access  panels  (for  low  vulnerability)  vs.  few  access 
panels/close  placement  of  components  (high  maintainability).  The  basis  for  these  early  concept  design 
tradeoff  studies  are  models/simulations.  They  are  used  to  assess  proposed  designs  against  the  user’s 
requirements.  Logistics  requirements,  based  on  the  ORD  and  Logistics  Support  Analysis  (LSA)  are 
included  in  tradeoff  decisions  using  high  level  logistic  models.  These  models  are  based  upon  past  experi¬ 
ence  with  similar  systems.  The  level  of  logistic  tradeoff  (as  with  other  disciplines  tradeoffs)  depends 
upon  a  risk  analysis  performed  against  the  system  specification.  The  other  cross-functional  disciplines 
(safety,  human  factors,  and  climatic)  may  also  be  included  in  tradeoffs,  but  to  a  lesser  extent  during  this 
early  phase. 

A2. 1 . 1 .  Primarily,  the  type  of  testing  required  during  the  conceptual  design  phase  for  a  new  system  is 
measurement  testing  (primarily  wind  tunnel  testing,  materials  testing,  propulsion  concept  testing,  and 
observability  testing)  -  to  provide  data  for  high  level  aero  models,  not  to  validate  the  models  (not 
enough  of  the  proposed  system’s  hardware  is  available  to  test).  A  test  matrix  must  be  developed 
which  ensures  data  will  be  gathered  at  the  necessary  conditions  (Mach  number,  density,  azimuth  and 
elevation,  etc.)  and  can  be  scaled  to  the  necessary  conditions  with  the  required  confidence. 

A2. 1 .2.  Figure  A2.1.  shows  the  primary  test  resources  used  during  the  concept  phase  and  the  level  of 
integration  (overlap)  between  the  three  A-P-A  functional  categories/disciplines.  The  6-step  test  pro¬ 
cess  introduced  in  the  main  body  of  this  manual  is  followed  regardless  of  what  resource  is  utilized  or 
what  phase  of  system  maturity  the  aircraft  is  in.  The  contractor  performs  nearly  all  the  testing  during 
conceptual  design  with  government  review.  The  contractor  may  utilize  their  own  facilities  or  govern¬ 
ment  facilities,  depending  upon  availability  and  capabilities. 
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Figure  A2.1.  Primary  Test  Resources  &  Integration  Level  During  Concept  Development. 
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A2.2.  Design  Reflnement  (pre-component).  Once  the  aircraft’s  overall  design  has  been  determined,  the 
next  step  is  to  develop  the  components.  However,  before  components  can  be  designed,  the  system/sub¬ 
system  in  which  they  function  must  be  designed  to  determine  the  individual  component’s  detailed  perfor¬ 
mance  requirements.  Therefore,  between  the  high  level  concept  design  and  the  component  development 
phase,  an  enormous  design  effort  is  undertaken  by  the  contractor  to  solidify  and  refine  the  aircraft’s  sys¬ 
tems/subsystems  conceptual  designs.  This  design  refinement  process  is  very  complex  and  the  details  are 
outside  the  scope  of  this  document  and  may  be  found  in  system  engineering  documents  (such  as 
MEL-STD  973,  Configuration  Management).  For  the  purposes  of  this  manual,  the  design  refinement  pro¬ 
cess  follows  a  work  breakdown  type  structure  and  dissects  the  weapon  system’s  general  requirements  to 
lower,  more  detailed  requirements  (accomplished  through  functional  and  allocated  baseline  -  see  Figure 
2.1.  and  refer  to  acquisition  course  documents  such  as  DOD  5000.52-M  for  details). 

A2.2.1.  Design  refinement  is  achieved  by  implementing  the  test  process  using  a  mix  of  prototype 
testing  and  modeling/simulation  to  provide  feedback  to  the  designers  (primarily  modeling/simula¬ 
tion).  Figure  A2.2.  illustrates  this  point.  As  during  the  previous  section  (concept),  during  this  early 
phase  of  the  system’s  maturity,  numerous  tradeoffs  occur  within  and  between  functional  categories/ 
disciplines. 
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Figure  A2.2.  Design  ReHnement  Process. 
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A2.1.1.  During  design  refinement  its  important  the  government  single  manager  (SM)/System  Pro¬ 
gram  Office  (SPO)  manager  and  contractor  keep  logistics,  safety,  and  human  factors  concerns  in 
mind.  System  attributes  such  as  reliability,  maintainability,  human  interfacing,  etc.,  must  be  designed 
into  the  system  (from  the  ORD  and  carried  through  to  system  specifications.  Integrated  Logistics  Sup¬ 
port  Plan  (ILSP),  etc.)  and  cannot  be  added  after  manufacturing  without  great  cost,  time,  and  risk  to 
the  program. 

A2.2.3.  Although  more  testing  (in  wind  tunnels  and/or  labs)  is  conducted  during  design  refinement 
than  during  concept  exploration,  the  majority  of  work  still  involves  developing  and  utilizing  models/ 
simulation  to  develop  the  system.  Details  and  examples  of  models/sims  used  by  each  A-P-A  disci¬ 
pline  are  included  within  each  specific  A-P-A  discipline’s  template  in  section  3.2.  The  tests  conducted 
in  labs  normally  involve  testing  new  technologies  (using  prototypes)  which  have  been  proposed  in  the 
system’s  design.  These  labs  could  either  be  contractor  or  government  operated. 

A2.2.4.  By  the  conclusion  of  acquisition  phase  0  and  at  milestone  1,  the  basic  air  vehicle  shape,  size, 
and  weight  has  been  determined,  as  well  as  the  overall  system’s  architecture  (federated,  integrated,  or 
integrated  suite).  The  A-P-A  disciplines  begin  to  solidify  their  system’s/subsystem’s  design  and  archi¬ 
tectures  based  on  the  aero  foundation  and  the  overall  system  architecture.  Preliminary  failure  modes 
and  effects  analysis  (FMEA)  is  performed  to  assess  system  failure  degradation  modes  and  failure 
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probabilities  against  the  ORD,  specifications,  and  MIL-STDs.  Preliminary  Live  Fire/vulnerability 
assessments  are  performed  using  models  and  contribute  to  design  tradeoff  decisions. 

A2.2.5.  The  A-P-A  disciplines  develop  more  models/simulations  and  build  upon  existing  ones  to  cre¬ 
ate  more  elaborate  models/simulations  as  the  system  develops  (i.e.,  models  are  rarely  discarded  as  the 
system  matures,  instead,  they  mature  with  the  system).  Besides  being  used  to  refine  the  design,  these 
early  models/simulations  also  serve  as  a  core  set  to  be  used  in  successive  phases  of  the  weapon  sys¬ 
tem’s  maturity  to  predict  test  results  (used  during  application  of  the  test  process  during  component, 
subsystem,  and  system  testing). 

A2.2.6.  In  this  early  phase  of  a  system’s  development,  prior  to  testing,  only  limited  model  verification 
can  be  achieved  (exploring  leading-edge  technologies  with  limited  test  data).  Fortunately  A-P-A 
models  are  largely  (though  not  exclusively)  based  upon  scientific  first  principles  and  laws  of  physics 
and  therefore,  the  core  equations  are  fairly  common  throughout  industry.  Avionics  models  however, 
are  based  more  on  a  mix  between  physics  and  logic  equations.  The  differences  between  various  mod¬ 
els  arise  from  the  values  used  for  the  variables  and  constants  contained  in  the  equations  because  these 
are  normally  derived  from  tests.  So  at  this  early  stage  of  model  development,  best  guesses  and  past 
experience  data  are  used  to  choose  the  variable/parameter  values. 

A2,3.  Component  Development  Phase.  Once  the  initial  A-P-A  component  and  subsystem  require- 
ments/de-signs  have  been  determined  (from  functional  and  allocated  baselines),  component  development 
begins.  The  basic  thrust  of  component  testing  is  to  ensure  the  components  perform  within  subsystems  as 
designed.  The  test  process  is  applied  to  prototype  components  and  actual  components  using  various  lev¬ 
els  of  test  resources  (models/simulations  through  open-air  testing).  Figure  A2.3.  shows  the  primary  test 
resources  used  during  component  development  and  level  of  integration/interfacing  between  the  three 
A-P-A  functional  categories/disciplines. 
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Figure  A2.3.  Primary  Test  Resources  &  Integration  Level  During  Component  Development. 
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A2.3.1.  Little  integration  and  tradeoff  occurs  between  A-P-A  disciplines  during  component  develop¬ 
ment  because  the  majority  of  components  do  not  interface  with  subsystems  outside  the  subsystem  in 
which  they  reside.  During  component  development  close  coordination  must  occur  within  the  func¬ 
tional  categories/disciplines  to  refine  the  systems/subsystems  designs  in  which  the  components  reside. 
Thus  ensuring  the  components  and  associated  subsystems  form  an  integrated  design  occurred  during 
the  design  refinement  process. 

A2.3.2.  The  test  resources  chosen  for  a  given  component  depends  upon  factors  such  as  cost,  design 
risk,  integration  requirements,  maturity  of  system  in  which  it  is  embedded,  simulation  fidelity/vari¬ 
ables,  and  schedule.  Component  test  results  are  compared  with  expected  results  from  system  and  sub¬ 
system  models  which  were  also  used  to  refine  the  overall  design.  Figure  A2.4.  shows  how  the  test 
process  is  applied  during  component  development. 
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Figure  A2.4.  Component  Development  Process. 


A2.3.3.  Generally  component  development  is  subcontracted  out  by  the  prime  contractor.  Obviously 
both  the  prime  and  subcontractors  must  work  closely  to  establish  component  requirements,  design 
components,  and  test  the  components  to  ensure  they  function  correctly  as  a  unit  (the  subsystem).  A 
goal  for  all  concerned  parties  (government,  and  contractors)  is  to  use  off-the-shelf  components  or 
slight  modifications,  in  which  case,  component  testing  is  dramatically  reduced.  For  state-of-the-art 
technologies  or  designs  with  stringent  requirements,  the  components  must  be  developed  from  scratch. 

A2.3.4.  Primarily,  the  models  used  during  component  development  are  the  same  subsystem  models 
used  to  refine  the  design.  Components  are  modeled  within  these  subsystem  models  and  represent 
component  performance  to  the  required  fidelity.  Normally,  the  A-P-A  components  are  modeled 
within  subsystem  and  system  models  and  do  not  have  their  own  unique  models  because  the  perfor¬ 
mance  requirements  for  an  A-P-A  component  can  usually  be  ascertained  directly  from  these  higher 
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level  models.  Stand  alone  component  models  are  created  if  the  component  requirements  push  the 
available  state-of-the-art  (high  risk)  or  must  very  accurately  represent  the  component.  For  stand  alone 
models,  comprehensive  component  testing  is  needed  to  ensure  high  model  fidelity. 

A2.3.5.  As  part  of  post-test  analysis,  test  results  are  compared  to  predicted  results  (from  models)  to 
ensure  components  function  as  designed  and  meet  requirements.  In  those  cases  where  the  component 
does  not  meet  requirements  or  the  design  does  not  meet  requirements,  corrective  action  is  required. 
What  corrective  action  is  required  is  dependent  upon  the  discrepancy,  the  system  contract,  and  system 
maturity.  As  stated  previously,  the  contractor  conducts  the  majority  of  testing  for  acquisition  phases 
O-I.  Therefore,  during  early  stages  of  system  development,  the  government  should  keep  a  log  of  noted 
problems  (using  the  watch-items  described  in  Air  Force  Technical  Order  (TO)  00-35D-54)  and  not 
submit  formal  deficiency  reports  (DR)  unless  determined  by  government  to  be  crucial.  To  adequately 
document  and  correct  the  deficiency/discrepancy,  a  formal  deficiency/discrepancy  is  required.  Per 
AFI 99-101  and  AFI 99-102,  the  government  SM  shall  ensure  the  contractor  has  an  adequate  discrep¬ 
ancy  reporting  system  established  prior  to  contractor  performed  component  testing  (which  can  easily 
transfer  data  or  directly  interface  with  the  government  DR  system)  and  the  government  has  a  DR  sys¬ 
tem  established  in  accordance  with  (LAW)  TO  00-35D-54  prior  to  government  performed  testing. 

A2.3.6.  As  stated  earlier  and  in  Figure  A2.3.,  the  facilities  used  to  conduct  component  testing  for  the 
A-P-A  disciplines  depends  upon  many  factors,  but  generally  component  testing  for  a  new  total  system 
development  is  conducted  with  models  and  simulations  Measurement  Facilities  (MF),  System  Inte¬ 
gration  Laboratories  (SIL),  and  limited  testing  in  Hardware-In-The-Loop  (HITL)  facilities.  Gener¬ 
ally,  many  of  these  types  of  facilities  are  owned  and  operated  by  contractors/subcontractors  and  the 
actual  tests  are  conducted  by  the  contractor/subcontractors  with  government  oversight  and  review  of 
test  results.  Details  regarding  the  various  facility  types  used  for  the  three  A-P-A  disciplines  are 
included  in  section  3.2  and  Attachment  3  of  this  manual. 

A2.3.7.  During  component  development,  the  common  thread  disciplines  (logistics,  human  factors, 
safety,  etc.)  must  ensure  the  components  meet  their  requirements.  Just  like  the  A-P-A  system  disci¬ 
plines,  they  must  compare  component  test  results  against  models  and  either  update  the  best  guess 
models,  or  influence  a  design  change.  The  level  of  influence  the  common  thread  disciplines  have  on 
the  design  engineers  depends  largely  upon  the  requirements  (i.e.,  high  reliability,  maintainability,  and 
availability  (RM&A)  ORD  requirements  have  a  stronger  influence  than  relatively  trivial  require¬ 
ments).  These  common  thread  disciplines  generally  "piggy-back"  their  testing  during  SIL  and  HITL 
tests.  In  some  cases  these  common  thread  disciplines  shall  conduct  dedicated  tests  against  ORD 
derived  requirements  which  have  been  flowed  down  to  allocated  baseline  requirements  and  verify 
model  predictions  (i.e.,  a  component  durability  test  to  assess  how  many  times  the  component  cycles 
before  failure  -  these  results  are  aggregated  with  other  component  RM&A  results  to  compare  to  sys¬ 
tem  level  ORD  requirements).  Mock-ups  and  prototypes  for  the  flight  control  subsystem,  structure, 
airframe  systems,  avionics  package,  and  powerplant  are  built  as  precursors  to  actual  hardware  labs. 
Human  factors  and  safety  problems  tend  to  surface  when  tasks  (either  crew  tasks  or  maintenance 
tasks)  are  performed  using  the  mock-ups  (which  then  influence  a  design  change). 

A2.3.8.  The  culmination  of  aU  the  tests  conducted  on  a  given  component  (functional  tests,  durability 
tests,  etc.)  "qualifies"  the  component  for  the  next  phase  of  system  maturity,  subsystem  development. 

A2.4.  Subsystem  Development  Phase.  As  a  system  progresses  through  the  development  phases,  assess¬ 
ing  the  actual  system  performance  against  the  users  requirements  becomes  more  straightforward.  The 
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assessments  become  more  straightforward  and  more  confidence  is  gained  in  the  design  (assuming  the 
assessment  is  positive)  as  the  system  development  progresses  from  components  to  total  system.  The  test 
objectives  are  elevated  from  specification  levels  to  higher  levels  -  levels  approaching  the  ORD  require¬ 
ments  (i.e.,  a  user’s  aircraft  range  requirement  in  the  ORD  can  only  be  directly  assessed  after  the  aircraft 
is  built  and  flight  tested).  The  subsystem  phase  is  a  critical  phase  to  assess  how  well  the  overall  design 
compares  to  the  user’s  requirements  because  it  is  the  first  opportunity  to  test  at  a  level  near  the  user’s 
requirements  without  excessive  component  extrapolation/aggregation.  Also,  subsystem  models  have 
been  updated  to  reflect  the  component  test  results  which  increases  the  models  fidelities  (and  confidence  in 
their  predictions).  Figure  A2.5.  shows  the  subsystem  development  process.  Note  that  this  is  very  similar 
to  the  component  development  process  (Figure  A2.4.),  but  includes  more  direct  comparison  of  test  results 
to  user  requirements  (not  just  model  predictions). 


Figure  A2.5.  Subsystem  Development  Process. 


A2.4. 1 .  By  this  phase  in  a  system’s  development,  the  overall  system  design  and  the  LS A  database 
should  be  fairly  well  solidified.  Preliminary  TOs  drafting  begins  in  support  of  early  DT&E.  Sub- 
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system  design  changes  may  be  required  to  fix  problems  or  implement  improvements  discovered  dur¬ 
ing  component  and/or  subsystem  testing,  but  these  changes  must  account  for  resulting  schedule  and 
cost  impacts.  Care  must  be  taken  to  not  be  overly  optimistic  regarding  cost  and  schedule  impacts.  It 
is  better  to  fix  and  retest  early,  than  believe  analysis  alone  without  test  verification  and  find  out  fur¬ 
ther  in  development  that  the  fix  did  not  work  or  was  not  possible. 

A2.4.2.  Along  these  same  lines,  LFT  of  components  and  subsystems  (using  prototype  or  production 
representative  components)  should  occur  no  later  than  this  stage  of  development  to  ensure  adequate 
time  to  make  a  design  change  if  required  and  to  budget  for  the  cost  of  such  testing.  The  LFT  Law, 
Title  10,  United  States  Code  (U.S.C.),  Chapter  139,  Section  2366,  et.seq.,  states  LFT  must  occur  suf¬ 
ficiently  early  in  the  development  phase  to  allow  for  any  LFT  design  deficiency  to  be  corrected  before 
proceeding  beyond  low-rate  initial  production  (LRIP).  Also  at  this  point  in  a  systems  development, 
the  general  LFT  strategy  should  be  developed  (i.e.,  is  the  system  "a  covered  system"  under  the  defini¬ 
tion  of  the  law  and  will  full-up  LFT  occur  or  will  the  Air  Force  seek  a  waiver  to  the  law).  Vulnerabil¬ 
ity  models  are  used  to  develop  the  LFT  matrix,  form  the  basis  for  the  LFT  strategy,  reduce  the  number 
of  shots,  and  help  analyze/understand  test  results.  For  further  Air  Force  guidance  regarding  LFT,  see 
AFI 99-105. 

A2.4.3.  During  the  subsystem  development  phase,  the  number  of  new  models  created  begins  to 
taper-off,  but  model  sophistication/complexity  continues  to  increase  as  a  system  develops.  Similarly, 
test  sophistication/complex-ity  has  increased  from  that  during  component  development.  Generally 
speaking,  subsystem  testing  is  primarily  conducted  in  SILs  and  HITLs  as  shown  in  Figure  A2.6. 
These  facilities  assess  how  well  a  given  subsystem  (and  components)  function  against  design  require¬ 
ments  and  also  how  well  it  integrates  with  other  subsystems.  Detailed  FMEA  is  typically  performed 
during  this  phase  and  the  system  development  phase  using  these  facilities.  This  is  accomplished  by 
inducing  a  set  of  test  matrix  failures  (either  through  hardware  or  software)  and  observing  the  effect  on 
the  subsystem/system.  The  type  of  induced  failures  ranges  from  single  input/output  signals  within  a 
component,  single  component  failures,  to  multiple  component  failures.  The  effect  these  failures  have 
on  the  overall  system  is  very  dependent  upon  the  system  architecture.  Generally,  integrated  suite 
designs  (F-22)  are  more  robust  than  federated  designs  (F-4),  but  the  FMEA  and  system  integration  is 
more  complex  for  integrated  suites. 
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Figure  A2.6.  Primary  Test  Resources  &  Integration  During  Subsystem  Development. 
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A2.4.4.  During  subsystem  development,  integration/interfacing  becomes  testable  to  a  limited  extent. 
Similar  to  extrapolating  component  test  results  to  assess  how  well  they  meet  user  requirements,  sub¬ 
systems  testing  is  the  first  opportunity  to  directly  assess  systems  integration  without  excessive  extrap¬ 
olation.  The  government  must  ensure  the  contractor  produces  and  uses  Interface  Control  Documents 
(ICD)  as  a  vehicle  to  ensure  the  system  designers  adequately  account  for  subsystem  integration/inter¬ 
facing. 

A2.4.5.  Usually  subsystem  testing  is  planned  and  conducted  by  the  contractors,  with  limited  govern¬ 
ment  involvement  (over  the  shoulder  or  reviewing  test  results).  By  this  point  in  a  system  acquisition 
[i.e.,  demonstration/validation  (DemA'al)  or  engineering,  manufacturing,  and  development  (EMD) 
acquisition  phase],  the  government  RTO  should  have  been  designated.  The  Responsible  Test  Organi¬ 
zation  (RTO)  should  assist  the  SPO  in  observing  and  reviewing  tests.  This  gets  the  testers  involved 
early,  which  results  in  more  effective  testing  (and  a  more  effective  system)  in  subsequent  maturity  and 
acquisition  phases. 

A2.5.  System  Development  Phase.  The  goals  for  this  system  development  phase  are  to  first  ensure  the 
air  vehicle  is  safe,  then  verify  the  air  vehicle  performs  basic  functions  as  designed,  and  finally  to  obtain  a 
preliminary  assessment  of  the  system’s  effectiveness  and  suitability.  If  problems  regarding  effectiveness 
and  suitability  are  discovered  during  this  test  phase,  corrective  action  will  be  taken,  but  a  comprehensive 
assessment  of  the  user’s  requirements  is  made  during  the  next  phase  of  the  weapon  system’s  development 
-  Integrated  System  testing. 

A2.5.1.  Usually  during  this  phase,  issues  regarding  the  air  vehicle’s  instrumentation  system  surface. 
Well  before  this  phase,  the  end-to-end  instrumentation  system  (measurement,  to  telemetry,  to  data 
storage)  should  have  been  designed  and  incorporated.  Typically,  instrumentation  is  a  large  effort  and 
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may  involve  dedicated  test  air  vehicles  which  may  or  may  not  be  retro-fitted  after  test  completion  to 
match  production  specifications.  Regardless,  sufficient  allowances  must  be  made  in  the  Test  and 
Evaluation  Master  Plan  (TEMP)  budget  design  to  account  for  instrumentation  requirements.  Things 
to  keep  in  mind  include:  data  sampling  rates  (to  ensure  necessary  data  are  not  aliased),  mux-bus  inter¬ 
ference  problems,  telemetry  bandwidth,  data  compression  algorithms,  and  onboard  data  tape  opera¬ 
tion.  Most  current  aircraft  programs  rely  upon  collecting  data  via  the  aircraft’s  own  MIL-STD-1553 
buses. 

A2.5.2.  During  the  system  development  phase  various  subsystems  and  components  come  together  to 
form  the  air  vehicle.  Actual  subsystem  components  are  installed  within  and  on  aircraft  structural 
components,  which  are  then  mated  to  form  the  air  vehicle.  At  each  step  of  assembly,  the  component 
installation  is  formally  verified  and  validated  against  drawings  by  inspectors  (contractor  and  govern¬ 
ment  qualification  engineers).  Then  are  rigorously  tested  against  functional  test  procedures  written  by 
the  design  engineers  for  functionality  using  special  test  equipment  (voltmeters,  pressure  sensors, 
torque  wrenches,  theodolites,  computer  interfaces,  etc.).  Once  components  are  linked  (in  hardware  or 
software)  to  form  subsystems,  the  subsystem  functionalities  are  formally  tested,  again,  through  highly 
controlled  and  regulated  procedures.  The  above  assembly  tests  are  commonly  referred  to  as  installa¬ 
tion  verification  and  validation  (IV&V)  and  will  not  be  addressed  to  further  extent  in  this  document  (a 
separate  process  outside  the  scope  of  this  document). 

A2.5.3.  A-P-A  system  testing  is  divided  into  distinct  phases:  pre-flight  testing  and  flight  testing.  The 
pre-flight  testing  is  specific,  rigorous  tests  to  ensure  subsystem  functionality  and  safety  prior  to  deliv¬ 
ery  of  the  aircraft  to  the  government.  The  "flight"  testing  phase  is  all  the  ground  and  flight  testing  per¬ 
formed  to  evaluate  the  capability  of  the  aircraft.  Ground  tests  are  typically  conducted  after  pre-flight 
tests  during  the  flight  test  phase  -  examples  include  braking  tests,  ground  handling,  and  logistics  tests. 
Data  from  both  the  pre-flight  and  flight  phases  are  used  to  verify  and  update  the  system  models. 

A2.5.4.  Figure  A2.7.  shows  the  system-level  test  process.  The  system  is  assessed  against  the  user’s 
requirements  during  flight  test,  during  actual  flights  or  during  ground  tests.  Pre-flight  tests  do  not 
have  strong  ties  to  the  user’s  requirements  and  are  mainly  conducted  to  ensure  the  aircraft  is  "ready" 
for  flight.  As  shown  in  Figure  A2.8.,  system-level  testing  is  conducted  primarily  using  Installed  Sys¬ 
tem  Test  Facility  (ISTF)  and  open-air  ranges  (OAR).  All  the  various  subsystems  function  within,  and 
are  tested  onboard,  an  integrated  environment  -  the  air  vehicle.  However,  to  reduce  the  number  of  test 
variables,  the  individual  subsystems  tests  are  conducted  somewhat  independently  (i.e.,  flight  controls 
has  a  series  of  unique  tests,  avionics  has  a  series  of  unique  tests,  propulsion,  etc.).  Not  until  the  next 
phase  of  system  maturity/development:  Integrated  System,  is  the  system  tested  using  integrated/mis¬ 
sion  level  tasks.  The  "common-thread"  disciplines  (logistics,  human  factors,  etc.)  can  be  realistically 
validated  on  the  air  vehicle  during  the  integrated  system  phase.  This  testing  is  identified  in  the  TEMP 
as  part  of  the  required  ground  tests  which  update  the  LS A  database  predictions  and  identify  RM&A 
achievements  or  shortfalls.  System  testing  is  usually  conducted  as  a  team  effort  involving  the  contrac- 
tor(s)  and  the  government  (DT&E  and  OT&E),  although  Air  Force  Operational  Test  and  Evaluation 
Center’s  (AFOTEC)  use  of  contractor  data  and  support  (the  system  under  test  contractor,  not  AFO- 
TEC  support  contractor)  is  restricted  per  10  U.S.C.  2399. 
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Figure  A2.7.  System  Level  Test  Process. 
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Figure  A2.8.  Primary  Test  Resources  &  Integration  Level  During  System  Development. 
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A2.5.5.  As  stated  in  the  previous  paragraph,  once  the  air  vehicle  has  been  assembled  and  IV&V’d,  a 
significant  number  of  pre-flight  tests  must  be  conducted  to  ensure  the  aircraft  is  flight  safe  and  "criti¬ 
cal"  systems  (those  necessary  to  sustain  flight)  function  properly.  These  pre-flight  tests  are  not  large 
scale  IV&V,  but  instead,  assess  whether  the  various  subsystems  function  as  an  integrated/cohesive 
unit  as  designed  -  the  air  vehicle.  These  tests  are  usually  conducted  by  the  contractor  in  ISTFs  or  on 
the  assembly  floor  using  special  test  equipment  and  are  monitored  by  the  government  (details  of  these 
tests  are  in  the  specific  A-P-A  disciplines  sections  later  in  this  document).  A  representative  (though 
not  exclusive)  series  of  pre-flight  tests  is  shown  in  Table  A2.1. 
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Table  A2.1.  Pre-Flight  Test  Phase  Examples. 


Ground  Test 

Purpose 

1. 

Functional  tests  of  A-P-A  subsystems:  hydrau¬ 
lic,  environmental  control  system  (ECS),  flight 
controls,  engines,  auxiliary  power  units  (APU), 
etc. 

Verify  installed  subsystems  function  and  inter¬ 
face  properly  within  installed  environment. 

2. 

Structural  ground  vibration  test  (GVT) 

Verify  structural  modes  of  the  air  vehicle  while 
configured  in  various  mass  property  conditions 
[center  of  gravity  (c.g.)  &  Fuel].  Requires  "soft 
supporting"  air  vehicle  off  gear  to  simulate 
flight  configuration. 

3. 

Ground  resonance  tests  (GRT)  and  limit  cycle 
tests 

Verify  adequate  control  system  gain  and  phase 
margins  (open  and  closed  loop).  Verify  interac¬ 
tion  between  structural  modes,  aerodynamics 
and  flight  control  system.  As  with  GVT,  air 
vehicle  must  be  "soft  supported". 

■ 

Electromagnetic  interference  (EMI)/electro- 
magnetic  compatibihty  (EMC)  Tests 

Avionics  and  electrical  equipment  on  the  air 
vehicle  are  operated  to  ensure  no  incompatibil¬ 
ities  exist  that  would  affect  flight  safety. 

5. 

Taxi  Tests  (Beginning  with  low  speed  and  pro¬ 
gressing  to  near  lift-off  speed) 

Last  series  of  ground  tests  to  verify  system  is 
ready  for  flight.  Exercise  system  while  aircraft 
is  in  motion. 

A2.5.6.  Finally,  after  all  the  years  of  planning  and  preparation  the  real  heart  of  system-level  testing  is 
conducted  -  open-air  flight  test.  Open  Air  Ranges  (OAR)  allow  flight  testing  under  a  controlled  envi¬ 
ronment  (to  reduce  number  of  test  variables  and  risk)  in  natural  climatic  conditions.  ISTFs  are  still 
used  during  open  air  testing  to  assist  with  mission  planning  and/or  to  investigate  problems  discovered 
during  open  air  testing. 

A2.5.7.  Flight  testing  begins  in  the  heart  of  the  flight  envelope  (altitude  and  airspeed)  and  the  heart  of 
individual  subsystems  functional  design  envelopes.  Along  the  way,  as  more  confidence  is  gained 
in-flight  safety  and  subsystem  functionality,  the  envelopes  are  expanded.  Not  all  subsystems  mature 
at  the  same  rate  and  subsystem  upgrades  occur  at  different  times.  Regression  testing  is  often  required 
when  these  upgrades  are  performed.  This  all  adds  significant  complexity  to  system-level  testing. 
Management  systems  must  be  created  to  track  the  air  vehicle’s  configuration  vs.  time,  flight  no.,  etc., 
and  systems  must  be  created  to  track  the  various  subsystem  envelopes.  Further  details  regarding 
A-P-A  system  flight  testing  may  be  found  within  each  specific  A-P-A  discipline’s  section  in  this  doc¬ 
ument. 

A2.5.8.  As  shown  in  Figure  A2.7.,  many  of  the  user’s  requirements  (range,  RM&A,  etc.)  will  be 
directly  tested  during  system  testing.  For  instance,  logistics  test  (LOG  TEST  as  defined  in  AFI 
99-101)  will  be  the  process  adopted  for  validating  RM&A  requirements.  Requirements  in  which  mis¬ 
sion  level  assets  are  integrated  with  the  test  air  vehicle  (for  example:  assessing  those  requirements 
which  integrate  Joint  Surveillance  Target  Attack  Radar  System  (J-STARS)  with  the  test  aircraft’s 
weapon  delivery)  are  assessed  during  the  next  total  integrated  system  phase.  In  instances  where  the 
air  vehicle  does  not  meet  the  user’s  needs  or  problems  need  to  be  documented  and  solutions  found,  a 
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formal  government  DR  will  be  written  as  required  by  AF  TO  00-35D-54.  The  SPO  will  then  work 
with  the  contractor  as  required  to  resolve  the  deficiency. 

A2.5.9.  Included  in  the  system  development  phase  (as  part  of  LOG  TEST)  is  preliminary  TO  valida¬ 
tion  and  verification  (TOV&V).  These  documents  mature  with  the  air  vehicle  and  testing  is  con¬ 
ducted  lAW  the  currently  verified/validated  TO  and  recommended  corrections  or  enhancements 
submitted  LAW  applicable  directives.  Maintenance  actions  should  be  performed  lAW  draft  TOs  and 
recommended  corrections  or  enhancements  be  explored/incorporated  into  final  versions. 

A2.5.10.  An  important  step  during  system  testing  is  OT&E  certification.  This  occurs  when  the  SM 
(in  consultation  with  DT&E  and  OT&E  representatives  and  HQ  USAF)  believes  the  system  is  ready 
to  transition  from  DT&E  to  dedicated  OT&E  (and  is  confident  the  weapon  system  will  pass  OT&E 
testing).  Details  of  this  process/step  may  be  found  in  AFI  99-101,  AFI  99-102,  and  USAF/TE’s 
OT&E  Certification  Templates. 

A2.5.11.  To  ensure  system  suitability  is  adequately  addressed,  structured  suitability  tests  will  be  con¬ 
ducted.  Logistics  tests  will  be  identified  in  the  TEMP  and  dedicated  time  and  resources  (manpower, 
test  articles,  support  equipment,  etc.)  identified.  In  the  past,  the  disciplines  typically  associated  with 
suitability  testing  (logistics,  "ilities",  human  factors,  safety,  etc.)  had  to  plan  their  test  in  a  piggy-back 
fashion.  This  may  be  called  for  in  some  instances  (to  save  time  and  money),  but  not  all.  Dedicated 
time  and  assets  must  be  set  aside  to  conduct  suitability  tests  and  ensure  the  delivered  air  vehicle  meets 
all  the  user’s  requirements  -  both  effectiveness  AND  suitability.  As  stated  in  previous  maturity  test 
phase  sections  of  this  document,  these  common  thread  disciplines  have  been  involved  with  the  air 
vehicle’s  development  from  the  beginning,  therefore  adequate  suitability  testing  should  be  conducted 
and  tied  closely  with  effectiveness  testing. 

A2.5.12.  By  this  point  in  system  development  (pre  LRIP  decision),  either  full-up  LFT  must  have 
occurred  [refer  to  LFT  Law,  Office  of  Secretary  of  Defense  (OSD)  guidance,  and  Air  Force  guidance 
for  details]  or  a  waiver  granted. 

A2.6.  Integrated  System  Test  Phase.  This  phase  of  system  development  is  concerned  with  testing  the 
weapon  system  in  an  environment  which  as  close  as  practical  replicates  the  environment  in  which  the 
weapon  system  will  operate.  Figure  A2.9.  shows  the  integrated  system  development  process.  The  goal 
for  this  phase  of  system  development  is  to  deliver  an  effective  and  suitable  weapon  system  to  the  user  with 

no  "gotchas"  or  "by  the  ways . ".  To  accomplish  this,  realistic  suitability  testing  must  be  performed  in 

addition  to  realistic  effectiveness  testing.  Examples  include,  but  are  not  limited  to:  exercising  the  ELSP, 
conducting  TOV&V,  conducting  survivability  testing  in  simulated  threat  environments  using  operational 
tactics,  performing  various  maintenance  tasks  (to  verify  LSA  predictions),  and  exercising  subsystems 
while  in  realistic  environmental  conditions.  Suitability  testing  evaluates  integration/interfaces  with  out¬ 
side  systems  (C3I  network,  ground-based  targeting  system),  and  conducting  mission  level  tasks  (bombing 
after  navigating  ingress  route,  live-fire  weapon  against  drones,  etc.).  This  testing  is  a  combination  of 
OT&E  and  DT&E  with  the  majority  of  the  testing  being  OT&E. 
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Figure  A2.9.  Total  Integrated  System  Development  Process. 


AFMAN99-110  3  JULY  1995 


113 


A2.6.1.  Figure  A2.10.  shows  the  resources  used  and  the  level  of  system  integration  required  during 
integrated  system  testing.  Similar  to  the  previous  development  phase  (system  testing),  integrated  sys¬ 
tem  testing  is  conducted  almost  exclusively  in  OARs  (with  less  rehance  upon  ISTFs  than  during  sys¬ 
tem-level  testing)  and  primarily  by  AFOTEC  (after  OT&E  test  readiness  certification).  This  usually 
requires  conducting  the  tests  at  ranges  (or  within  special  designated  areas)  which  approximate  opera¬ 
tional  environments.  These  ranges  include  threats  and  threat  surrogates,  operational  representative 
targets,  and  are  usually  instrumented  for  test  purposes. 

Figure  A2.10.  Primary  Test  Resources  &  Integration  Level  During  Total  Integrated  System 

Testing. 


A2.6.2.  Obviously,  now  the  system  is  performing  tasks  which  require  high  levels  of  system  integra¬ 
tion/interfacing  and  the  test  objectives  are  aimed  toward  mission-level  tasks  [aerial  delivery,  bomb¬ 
ing,  suppression  of  enemy  air  defenses  (SEAD),  etc.].  Therefore,  the  corresponding  models  used 
during  this  phase  of  development  are  mission-level  models  similar  to  those  during  concept  explora¬ 
tion.  Note  the  development  cycle  has  gone  full  circle  by  the  fact  that  the  integrated  system  develop¬ 
ment  focus  (mission-level  tasks)  is  aligned  very  closely  with  the  concept  development  focus  (mission 
level  tasks)  and  the  corresponding  high-level  ORD  requirements  (mission-level  tasks).  Also  note  that 
the  level  of  integration  required  during  development  and  testing  during  the  various  system  maturity 
phases  has  gone  full  circle.  Close  cooperation/coordination  between  and  with  subsystems  was 
required  during  concept  development  and  then  again  during  integrated  system  development. 

A2.7.  P^I  Major  Modification.  A  "modification"  is  a  change  to  a  system  (whether  for  safety,  to  cor¬ 
rect  a  deficiency,  or  to  improve  program  performance)  that  is  still  being  produced.  An  "upgrade"  is  a 
change  to  a  system  (whether  for  safety,  to  correct  a  deficiency,  or  to  improve  program  performance)  to  a 
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system  that  is  out  of  production.  A  "major  modification"  to  a  program  is  defined  as  a  modification  that  in 
and  of  itself  meets  the  criteria  of  acquisition  category  I  or  11  or  is  designated  as  such  by  the  milestone 
decision  authority.  Major  modifications  require  a  Milestone  IV  decision  unless  the  decision  to  modify 
results  from  one  of  the  alternatives  considered  as  part  of  the  Milestone  I  decision  process.  Upgrades  are 

part  of  the  milestone  0  decision  process. 

A2.7.1.  As  with  all  the  system  maturity  phases  (concept  through  fielding),  the  level  of  testing  is 
dependent  upon  the  specific  P^I/mod.  The  test  resources,  analysis,  and  the  T&E  pace  is  very  depen¬ 
dent  upon  the  exact  P^I/mod  requirement.  For  a  P^I/mod,  tradeoff  studies  are  more  straightforward 
than  with  a  new  systems  design  because  there  is  already  an  existing  system  with  which  to  assess  the 
proposed  designs  against  (the  number  of  unknowns  and  test  variables  is  drastically  reduced)  and  the 
integration  requirements  are  more  clearly  understood  early  in  the  study. 

A2.7.2.  If  the  P^I/mod  is  focused  toward  one  of  the  three  A-P-A  functional  categories/disciplines,  air¬ 
frame  for  example,  then  the  majority  of  development  and  testing  will  occur  within  that  functional  cat¬ 
egory/discipline.  For  example,  an  upgrade  to  an  airframe  system,  such  as  a  wing  upgrade  or  change  to 
outer  mold  line,  would  require  significantly  more  aerodynamic  testing  than  avionics  testing.  Other 
airframe  category  tests  may  be  required  (system  closed-loop  stability),  as  well  as  some  other  func¬ 
tional  category/discipline  development  and  testing  (change  to  systems  within  the  wing).  However, 
this  may  not  always  be  the  case  and  depends  on  factors  such  as  interface  requirements,  safety,  and 
design  risk.  Nonetheless,  the  test  process  side  of  the  T&E  cube  always  applies. 

A2.7.3.  As  stated  earlier,  the  test  resources  utilized  during  the  P^I/mod  program  is  dependent  upon 
the  exact  nature  of  the  P^I/mod.  Unlike  a  new  systems  development  where  generally,  T&E  starts  with 
modeling  and  simulation  measurement  facilities  and  ends  with  open-air  ranges,  P^I/mod  programs 

can  start  at  any  facility  level.  Component  testing  of  a  P^I/mod  could  be  conducted  on  a  flying  test  bed 
without  going  through  SILs,  HITLs,  etc.,  if  the  test  team  (SPO,  contraetor,  and  RTO)  deem  the  impact 

and  risk  is  minimal.  System-level  testing  a  P^I/mod  generally  is  conducted  similar  to  new  system  test¬ 
ing  described  above.  However,  pre-flight  tests  are  less  extensive  and  less  problems  are  encountered 

with  tracking  multiple  envelopes  and  configurations.  Like  new  system  testing,  P^I/modification  test¬ 
ing  primarily  occurs  in  OARs  and  ISTFs.  These  OARs  may  either  be  ranges  designed  specifically  for 
flight  test  (for  example,  Edwards  AFB  CA)  or  ranges  utilized  by  the  Air  Logistics  Centers  (ALC). 
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Attachment  3 
T&E  RESOURCES 


Introduction 

The  following  attachment  provides  brief  descriptions  and  points  of  contact  (POC)  for  government  test  and 
evaluation  (T&E)  facilities,  laboratory  facilities  used  for  research  and  Air  Logistics  Center  Depot  mainte¬ 
nance  infrastructure  resources  [associated  with  Airframe-Propulsion- Avionics  (A-P-A)  related  T&E]. 
The  laboratory  and  ALC  resources  are  listed  as  potential  capabilities  to  support  development  and  acquisi¬ 
tion  programs  if  T&E  facilities  are  not  available  or  adequate  to  meet  requirements.  The  facilities  are  cat¬ 
egorized  using  the  Reliance  T&E  Resource  Categories  (Measurement  Facilities  (MF),  System  Integration 
Labs  (SIL),  etc.).  Also  see  the  Test  and  Evaluation  Computer  Network  (TECNET)  (computer  hosted  list), 
Automated  Test  Resources  Information  System  (ATRIS)  Air  Force  Acquisition  Model  (AFAM),  Test 
Investment  Master  Plan  (TIMP),  Test  Capability  Master  Plan  (TCMP),  and  Test  Resource  Master  Plan 
(TRMP)  for  additional  information  regarding  Department  of  Defense  (DOD)  T&E  resources. 

Note:  Facilities  which  are  listed  in  T&E  resource  related  documents  (TISP,  TCMPs,  and  TRMP)  are  cat¬ 
egorized  using  the  common  reliance  categories  and  may  be  very  specific  to  a  given  Major  Range  and  Test 
Facility  Base  (MRTFB).  For  the  purposes  of  this  document  however,  many  small/very  specific  facihties 
are  not  listed,  but  instead,  are  captured  under  their  MRTFBs  reliance  category.  For  example,  the  weight 
and  balance  facility  located  at  Edwards  Air  Force  Base  (AFB),  CA  is  a  MF  and  is  not  specifically  listed  in 
this  attachment,  however  its  function  is  associated  with  the  flight  test  category  under  open-environment 
testing  -  (the  facility  measures  parameters  associated  with  flight  testing  of  air  vehicles). 

Detailed  facility  descriptions  for  selected  T&E  organizations  can  be  found  in  the  following  documents. 
This  information  can  be  used  to  supplement  the  resource  summary  data  provided  in  this  attachment. 

-Arnold  Engineering  Development  Center  (AEDC) ,  Test  Facilities  Handbook  (Thirteenth  Edition)  May 
1992.  Contact:  AEDC/DOCS,  Arnold  AFB  TN  37389-5000,  DSN:  340-6653. 

-Wright  Laboratory  (WL),  Research  and  Development  Facilities  Handbook,  WL-TR-92-0004,  Aug 
1992.  Contact:  WL/DOT,  Wright  Patterson  AFB  OH  45433-6523,  DSN:  785-4404. 

—Air  Logistics  Centers  (ALCs),  Ground  Test  and  Evaluation  Capabilities,  1  June  1994.  Contact: 
AFDTC/XRM,  Eglin  AFB  FL,  DSN:  872-4190. 

—84th  Test  Squadron,  Radar  Test  Facility  User’s  Guide,  Tyndall  AFB  FL.  Contact:  84TS/DOR,  Tyndall 
AFB  FL  82403-5435,  DSN:  523-3376. 

—Rome  Laboratory,  Facilities  Register,  RL-TR-93-91,  April  1993.  Contact:  RL/ERPE,  Griffiss  AFB 
NY  13441-4505.  DSN:  587-3282. 

-AFFTC  Range  User’s  Guide,  Edwards  AFB  CA.  Contact:  412TW/TSR,  DSN:  527-3050. 

— AFDTC  Technical  Facilities  Manuals  -  Vol  I,  H,  III  (Class).  Contact:  46  TW/XP,  DSN:  872-5307 

It  is  recommended  that  the  user  first  contact  the  SFTC  office  for  assistance  in  test  planning  and  obtaining 
recommendations  for  T&E  resources.  The  specific  facility  points  of  contact  listed  in  this  attachment  are 
provided  to  assist  the  user  in  obtaining  detailed  data  on  test  capabilities. 

Modeling  and  Simulation 
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These  resources  are  used  in  every  stage  of  a  system’s  development  to  help  design,  predict,  and  analyze 
tests.  Generally,  during  the  concept  phase,  modeling  and  simulation  (M&S)  form  the  basis  for  design 
tradeoff  studies  and  are  at  a  high  level  (for  example,  threat  scenarios,  and  environments).  As  the  system 
develops  from  components  to  the  fielded  system,  M&S  can  be  used  to  provide  generic  models  for  compo¬ 
nents,  interfaces,  etc.,  for  use  in  system  design  and  test.  M&S  develops  with  the  system  (using  test  results 
to  update  and  refine)  and  progresses  from  low  level/extremely  detailed,  to  more  high  level,  culminating  in 
the  Operational  Requirements  Document  (ORD)/Cost  and  Operational  Effectiveness  Analysis  (COEA) 
level  once  again  (Note:  the  requirements  have  gone  full  cycle  from  high  level,  to  low  level,  then  back  to 
high  level).  Increased  reliance  upon  M&S  is  encouraged  during  developmental  test  and  evaluation 
(DT&E)  and  operational  test  and  evaluation  (OT&E). 

A3.1.  Airframe: 

A3.1.1.  Aerodynamics  and  Performance  Models.  [Including  computational  fluid  dynamics 

(CFD)] 

A3. 1.1.1.  AEDC  -  External  Flow  Field  Models  of  Aircraft  with  and  without  external  stores  or 
stores  in  internal  bays.  Contact:  AEDC/DOF,  DSN:  340-7721. 

A3. 1.1. 2.  ASC  -  Provides  air  vehicle  Aero/Performance  estimates  including  external  aero  (e.g., 
lift  and  drag),  internal  aero  (e.g.,  installed  engine  performance  and  airframe/engine  compatibility, 
and  air  vehicle  performance  (e.g.,  mission  and  point  performance).  Contact:  ASC/ENFT,  Flight 
Mechanics  Branch,  DSN:  785-5503. 

A3. 1.1. 3.  AEDC  -  Inlet  ducting  flow  field  models.  Contact:  AEDC/DOF,  DSN:  340-7721. 

A3. 1.1. 4.  Wright  Lab  -  CFD  methodology  development  with  emphasis  on  increased  computa¬ 
tional  speed  while  remaining  robust  with  reasonable  accuracy.  Contact:  WL/FIMC,  DSN: 
785-4305. 

A3.1.2.  Structural  and  Finite  Element  Modeling: 

A3. 1.2.1.  AEDC  -  General  Purpose:  ANSYS  (mainframe),  NASA  Structural  Analysis  (NAS- 
TRAN)  experience,  ALGOR  (PC),  Specialty:  FLOMODL  (PC  based  steady  state  or  transient 
flow  network  code).  Contact:  AEDC/DOF,  DSN:  340-5305. 

A3. 1.2.2.  ASC  -  Provides  air  vehicle  structural  analysis  and  evaluation.  Contact:  ASC/ENFS, 
Structures  Branch,  DSN:  785-3330. 

A3. 1.2. 3.  Wright  Lab  -  Structural  Modeling  Tools  &  Analysis.  Contact:  WL/FIB,  DSN: 
785-5200. 

A3. 1.2. 4.  AFDTC  -  Structural  and  Finite  Element  Modeling.  Contact:  460G/0GME  DSN: 
872-3017. 

A3. 1.2.5.  Seiler  Lab  -  Numerical  Modeling  and  experiments  on  unsteady  aerodynamics  including 
3-D  unsteady  vortex  dynamics,  unsteady  boundary  layer  transition/separation,  and  active  flow 
control  and  free  shear  layer  instability  analysis.  Contact:  Seiler  Lab  DSN:  259-3120. 

A3.1.3.  Ground-Based  Piloted  Simulation: 
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A3. 1.3.1.  AFFTC  -  T&E  Mission  Simulator  (TEMS).  Provides  flight  test  programs  with 
real-time  piloted  (fixed  and  motion)  simulation  of  flight  dynamics  and  aircraft  systems.  Contact: 
412  TW/EWW,  DSN:  527-0840. 

A3. 1.3.2.  Wright  Lab  -  Flight  simulation  lab  provides  multi-aircraft  full  mission  level  (avionics, 
control  weapon,  threat)  piloted  simulation  in  40’  dome,  20’  moving  base  dome,  and  multiple 
manned  combat  stations  with  360  degrees  field-of-view  (FOV)  -threat  or  friendly.  Contact:  WL/ 
FIGD,  DSN:  785-4690. 

A3.1.4.  In-flight  Simulation: 

A3. 1.4.1.  Wright  Lab  -  Total  In-flight  Simulator  (TIFS).  A  modified  NC-131H  aircraft  owned  by 
Wright  Labs  and  operated  by  Calspan  Corp.  which  is  configured  with  simulation  cockpit  (can  be 
customized  to  match  test  aircraft)  in  addition  to  flight  cockpit.  Onboard  digital  computers  allow  6 
degree-of-freedom  rigid  body  simulation  of  an  aircraft’s  control  system/flying  qualities.  In-flight 
simulation  give  pilots  cues  not  available  in  ground  based  simulations.  Contact:  WL/Flight 
Dynamics  Directorate,  DSN:  785-3853.  The  TIFS  can  also  be  configured  with  a  radome,  in  place 
of  the  evaluation  cockpit,  to  house  a  variety  of  radar  and  Forward  Looking  Infrared  (FLIR)  sys¬ 
tems  for  flight  testing.  In  this  configuration,  the  aircraft  can  be  flown  from  a  pilot  station  in  the 
cargo  compartment  of  the  aircraft.  Contact:  WL/FIGD  DSN:  785-3853. 

A3. 1.4.2.  Wright  Lab  -  Variable  Stability  In-flight  Simulator  Test  Aircraft  (VISTA).  A  modified 
F-16D  Block  30,  Peace  Marble  configuration  owned  by  AF  and  operated  by  Calspan  Corp.  The 
onboard  digital  computers  make  the  aircraft  capable  of  simulating  the  flying  characteristics  of 
high  performance  aircraft.  Contact:  WL/Flight  Dynamics  Directorate,  DSN:  785-3853. 

A3.1.5.  Survivability/Vulnerability  Models: 

A3. 1.5.1.  Wright  Lab  -  The  Survivability/Vulnerabihty  Information  Analysis  Center  (SURVIAC) 
has  models  that  address  engagement  functions  such  as  exposure,  detection,  tracks,  launch  and 
guidance,  damage  assessment  and  failure  analysis.  Contact:  WL/FIVS/SURVIAC,  DSN: 
785-4840. 

A3. 1.5.2.  ASC  -  The  Joint  Modeling  and  Simulation  System  (J-MASS)  provides  a  simulation 
support  environment  (SSE)  and  modeling  library.  The  SSE  enables  the  user  to  create  models,  con¬ 
figure  scenarios,  execute  simulations  and  analyze  results.  The  library  contains  models  of  weapon 
systems/components,  threats,  and  environments.  Contact:  ASC/RWWW,  DSN:  785-3969. 

A3.1.6.  Weapons  Integration: 

A3. 1.6.1.  AEDC  -  Safe  Separation  Models.  Contact:  AEDC/DOFA,  DSN:  340-7721. 

A3. 1.6.2.  AEDC  -  Ballistic  accuracy,  bombing  algorithm  and  separation  coefficients.  Contact: 
AEDC/DOFA,  DSN:  340-7721. 

A3. 1.6.3.  AFFTC  -  System  Accuracy  Evaluation  Software  System.  Contact:  412TW/TSVW, 
DSN:  525-9094. 

A3. 1.6.4.  ASC/SK  -  The  SEEK  EAGLE  Office  at  Eglin  AFB  has  safe  separation  models.  Con¬ 
tact:  AFSEO/TA  DSN:  872-9052. 

A3. 1.6.5.  AFDTC  -  PRIMES  and  the  GWEF-PRIMES  Links  have  weapons  integration  capabil¬ 
ity.  Contact:  dbTW/TSWW  DSN:  872-9354. 
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A3.1.7.  Air  Vehicle  Subsystem  Models: 

A3. 1.7.1.  ASC  -  Provides  subsystems  integration/performance  estimates  [e.g.,  environmental 
control  system  (ECS),  fuel,  and  landing  gear].  Contact:  ASC/ENFA,  Air  Vehicle  Subsystem 
Branch,  DSN:  785-5178. 

A3.2.  Propulsion: 

A3.2.1.  Performance,  Specification,  and  Transient  Decks:  (computer  models  of  engine) 

A3.2.1.1.  AEDC  -  Models  are  used  in  aircraft  performance  predictions  (calculates  thrust  at  given 
ambient/flight  condition),  and  used  in  engine  development/design.  The  models  are  developed  by 
and  are  hosted  at  engine  contractors  in  conjunction  with  DOD  test  organizations.  For  decks  used 
during  flight  test.  Contact:  AFFTC/DOEF,  DSN:  525-9017.  For  decks  used  during  development. 
Contact:  AEDC/DOP,  DSN:  340-5305. 

A3 .2. 1.2.  AEDC  -  Developed  propulsion  modeling  software  called  "ATEST".  Uses  thermody¬ 
namic  cycle  equations  to  simulate  steady-state  engine  performance.  If  dynamic  control  system 
information  is  available,  it  can  also  model  dynamic  performance  for  throttle  motions  and  airstarts. 
Contact:  AEDC/DOP,  DSN:  340-5305. 

A3.2.1.3.  Wright  Lab  -  Conducts  engine  thermodynamics,  turbine  performance,  and  compressor 
analysis  modeling  in  support  of  research  and  conceptual  propulsion  systems.  Contact:  WL/POT, 
DSN:  785-2331. 

A3.2.1.4.  ASC  -  Provides  engine  performance  stability  estimates.  Contract:  ASC/ENFP,  Propul¬ 
sion  Systems  Branch,  DSN:  785-9595. 

A3.2.1.5.  AEDC  developed  propulsion  modeling  software,  “DYNTECC”  is  a  one-dimensional, 
stage-by-stage  compression  system  mathematical  model  which  is  able  to  analyze  any  generic  sys¬ 
tem.  It  has  the  capability  to  analyze  post-stall  behavior  as  well  as  predict  the  onset  of  compression 
system  instability.  Stability  limit  analysis  can  be  conducted  for  single-spool  and  dual-spool  sys¬ 
tems  with  and  without  distortion.  Contact:  AEDC/DOP  DSN:  340-5305. 

A3.2.1.6.  AEDC  supported  propulsion  modeling  software  “NPARC”  Navier-Stokes  flow  simula¬ 
tion  program  is  used  to  calculate  the  properties  of  a  fluid  flow,  based  on  specific  boundary  sur¬ 
faces  and  flow  conditions.  It  has  been  used  on  aerospace  propulsive  applications  as  diverse  as 
supersonic  and  hypersonic  inlet  design,  rocket  nozzle  failure  analysis,  turbine  engine  exhaust 
mixer  design,  missile  nose  cone  analysis,  instrumentation  probe  design,  and  ducted  flow  analysis. 
Contact:  AEDC/DOP  DSN:  340-5305. 

A3.2.1 .7.  AEDC  supported  integrated  modeling  software  based  on  the  GENeralized  Environment 
for  the  Simulation  of  Integrated  Systems  (GENESIS)  for  the  prediction  of  overall  integrated  vehi¬ 
cle  performance.  Integrated  vehicle  performance  is  based  on  sub-models  of  the  airframe,  propul¬ 
sion,  and  control  system  performance.  Specific  vehicle  simulations  (i.e.,  F-16,  F-15/SMTD, 
Mark-80  Bomb)  are  implemented  at  AEDC.  Contact  AEDC/DOP  DSN:  340-5305. 

A3.2.2.  Aerodynamic  Models:  (also  see  Airframe  M&S  in  this  attachment) 

A3.2.2.1.  AEDC  -  Stage  stacking,  cascade,  quasi  3D  modeling.  Contact:  AEDC/DOP,  DSN: 
340-5305. 


AFMAN99-110  3  JULY  1995 


119 


A3.2.2.2.  Wright  Lab  -  Propulsion  system  aerodynamic  modeling  for  research  and  conceptual 
designs.  Contact:  WL/POT,  DSN:  785-2331. 

A3. 2.2.3.  ASC  -  Provides  engine  performance  and  airframe/engine  integration  and  compatibility 
analyses.  Contact:  ASC/ENFP,  Propulsion  Systems  Branch,  DSN:  785-9595. 

A3.2.3.  Structural  Models:  (also  see  Airframe  M&S  in  this  attachment) 

A3. 2. 3.1.  AEDC  -  Finite  element  models  on  mainframe  or  PC.  Contact:  AEDC/DOP,  DSN: 
340-5305. 

A3. 2.3.2.  ASC  -  Provides  engine  structural  analyses.  Contact:  ASC/ENFP,  Propulsion  Systems 
Branch,  DSN:  785-9595. 

A3.2.3.3.  Wright  Lab  -  Turbine  Engine  Structural  Analysis  Center  for  Research  and  Conceptual 
Design.  Contact:  WL/POT,  DSN:  785-2331. 

A3.2.4.  Simulation  of  Inlets,  Ducts,  and  Nozzle  Afterbodies: 

A3.2.4.1.  AEDC  -  CFD  Simulation  of  inlets,  ducts,  and  nozzle  afterbodies.  Contact:  AEDC/ 
DOFA,  DSN:  340-7721. 

A3.2.5.  Piping  Networks: 

A3.2.5.1.  AEDC  -  Mainframe  (ANSYS)  or  PC  (FLOMODL)  steady  state  or  transient  capability. 
Contact:  AEDC/DOP,  DSN:  340-5305. 

A3.2.6.  IR  Models: 

A3.2.6.1.  AEDC  -  Infrared  (IR)  subroutines  appended  to  performance  models.  Contact:  AEDC/ 
DOP,  DSN:  340-5305. 

A3.2.6.2.  AFDTC  -  Airframe  and  engine  signature  and  scene  data  reduction  and  repositories  are 
maintained  for  use  in  countermeasure  modeling.  Contact:  96CCSG/SCW  DSN:  872-2180. 

A3.3.  Avionics: 

A3.3.1.  In-flight  Simulations:  (see  Avionics  Flying  Test  Beds  at  end  of  this  attachment) 

A3.3.2.  Integrated  Avionics  Simulations: 

A3.3.2.1.  Wright  Lab  -  The  Integrated  Avionics  Laboratory  (lAL)  is  a  combined  avionics 
research,  development,  modeling  and  simulation  facility  that  provides  the  ability  to  accomplish 
research  and  independent  verification  of  avionics  concepts  and  systems.  Its  purpose  is  to  conduct 
real-time/non-real-time,  multi-spectral,  multi-disciplinary  experiments  and  analysis  in  the  areas  of 
integrated  avionics,  core  processing  architecture,  information  processing,  communication,  naviga¬ 
tion,  identification,  software,  hfe  cycle  support  and  machine  intelligence.  Contact:  WL/AAA, 
DSN:  785-5218. 

A3. 3.2.2.  Wright  Lab  -  The  Electronic  Combat  Simulation  Research  Lab  (ECSRL)  provides 
expertise  and  digital  tools  for  supporting  studies,  COEAs  and  technology  insertion  development 
for  the  J-MASS.  Contact:  WL/AAWA-1,  DSN:  785-4429. 


A3.3.3.  CNI  Related  M&S: 
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A3.3.3.1.  AFDTC  -  Central  Inertial  Guidance  Test  Facility  (CIGTF),  Holloman  AFB  NM.  This 
facility  performs  T&E  measurement  of  guidance  systems  (missiles  and  aircraft)  and  has  extensive 
knowledge  in  related  M&S.  Contact:  46TG/GD,  DSN:  349-2123. 

A3. 3.3.2.  Wright  Lab  -  The  Communications  Systems  Evaluation  Lab  (CSEL)  provides  a 
dynamic  evaluation  of  state-of-the-art  communications,  navigation  and  identification  (CNI)  sys¬ 
tems  in  the  integrated  electromagnetic  system  simulator  (ESS)  and  provides  computer  controlled 
threat  interference  signals.  Contact:  WL/AAAI,  DSN:  785-2766. 

A3.3.3.3.  Wright  Lab  -  The  Integrated  Test  Bed  (ITB)  provides  a  real-time  simulation  of  aircraft 
performing  an  operational  mission  allowing  evaluation  of  advanced  avionic  systems,  architecture 
and  components  across  a  broad  spectrum  of  performance  requirements.  Contact:  WL/AAAS, 
DSN:  785-4827. 

A3.3.4.  Embedded  Software: 

A3.3.4.1.  Wright  Lab  -  Embedded  Computer  Resources  Support  Improvement  Facility  provides  a 
capability  to  develop,  test  and  evaluate  new  technologies  designed  to  improve  support  capability 
of  avionics  embedded  system  software.  Contact:  WL/AAAF,  DSN:  785-3826. 

A3.3.4.2.  ASC  -  Embedded  Computer  Resources  -  Integrated  Engineering  and  Technical  Man¬ 
agement  data.  Contact:  ASC/EN  (CR),  DSN:  785-3656. 

A3.3.5.  Airborne  Radar  and  Fire  Control  Modeling: 

A3. 3. 5.1.  Wright  Lab  -  The  Radar  Analysis  and  Signal  Processing  Lab  (RASPL)  provides 
state-of-the-art  modehng,  analysis,  simulation  and  signal  processing  environment  for  conducting 
air-to-air  (A/A)  and  air-to-ground  (A/G)  radar  system  studies.  Contact:  WL/AARM,  DSN: 
785-6071. 

A3.3.5.2.  Wright  Lab  -  The  Fire  Control  Simulation  (FICSIM)  Facility  provides  analysis  of  fire 
control  system  performance  in  A/A  arena,  and  A/G  arena.  Detailed  modeling  of  radar  sensor  with 
airborne  radar  detection  (AIRADE)  model  and  electro-optic  sensors  in  the  Electro-Optical  Simu¬ 
lation  (EOSIM)  model.  Contact:  WL/AART,  DSN:  785-3215. 

Measurement  Facilities  (MF) 

These  provide  capabilities  to  measure  parameters  critical  for  system  design.  Generally,  they  are 
not  test  article  unique,  but  do  perform  a  specific  function  (for  example,  signature  measurement  or 
aerodynamic  forces).  A-P-A  examples  include:  wind  tunnels,  radar  cross  section  (RCS)  ranges, 
spin-tunnels,  and  the  electromagnetic  interference  (EMI)/electromagnetic  compatibility  (EMC). 

A3.4.  Airframe: 

A3.4.1.  Wind  Tbnnels: 

A3.4.1.1.  AEDC  -  The  Propulsion  Wind  Tunnel  (PWT)  Facility  includes  16  ft.  tunnels  for  testing 
the  aerodynamic  performance  of  full-scale  engine  installation,  large  aircraft  models  or  large  and 
full-scale  missiles  for  transonic  (16T)  and  supersonic  (16S)  operating  regimes.  A  4  ft.  transonic 
tunnel  (4T)  is  used  primarily  for  store  separation  testing.  Contact:  AEDC/DOFA,  DSN: 
340-7721. 
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A3 .4. 1.2.  AEDC  -  The  Von  Karman  Gas  Dynamics  Facility  is  comprised  of  aerodynamic  test 
units  to  permit  testing  of  relatively  large-scale  models  of  high  speed  aircraft  and  missiles  in  a 
Mach  number  range  extending  from  1.5  to  10.  Tunnel  A  is  supersonic  with  a  40x40  inch  test  sec¬ 
tion  and  a  Mach  number  range  of  1.5  to  5.5.  Tunnel  B  is  hypersonic  with  a  50  inch  diameter  and 
a  Mach  number  range  of  6  or  8.  Tunnel  C  is  hypersonic  with  a  50  inch  diameter  and  a  Mach  num¬ 
ber  10  and  supersonic  aerothermal  with  a  25  inch  diameter  and  Mach  number  4  or  8.  Contact: 
AEDC/DOFA,  DSN  340-7721. 

A3.4.1.3.  Wright  Lab  -  The  Flight  Dynamics  Laboratory  has  seven  research,  development  and 
test  wind  tunnels  capable  of  providing  measurement  facilities  for  airframe  T&E. 

-Subsonic  Aerodynamic  Research  Laboratory  (SARL).  This  is  a  10  by  7  ft.  subsonic  wind  tun¬ 
nel  able  to  produce  a  flow  speed  to  0.5  Mach  number.  The  SARL  provides  high  quality  subsonic 
flow  for  design  and  for  analytical  methods  validation  purposes. 

-Vertical  Wind  Tunnel  (VWT).  This  facility  is  a  12  ft.  diameter  open  jet  test  section  wind  tunnel 
capable  of  speeds  to  125  ft./sec  to  test  paracWes,  ejection  seats,  free  falling  bodies,  and  to  study 
low  speed  aerodynamics.  The  tunnel  has  even  been  utilized  to  train  paratroopers.  It  is  projected 
that  the  VWT  will  have  a  rotary  balance  support  to  allow  dynamic  testing  in  FY96. 

-Two  Foot  Water  Tunnel  (LWT).  A  low  cost  test  facility  to  provide  a  vivid  depiction  of  the  flow 
field  about  air  vehicles  for  design  and  analysis  purposes. 

-Two  Foot  Trisonic  Gasdynamics  Facility  (TGF).  A  variable  density  wind  tunnel  providing  a 
test  Mach  number  range  of  subsonic  through  Mach  3,  for  design  and  analysis  purposes.  The  den¬ 
sity  can  be  varied  from  0.25  to  1.5  atmosphere. 

-Mach  3  High  Reynolds  Number  Facility  (M3).  This  blow  down  facility  simulates  flight  at 
Mach  3  over  an  altitude  range  from  sea  level  to  30,000  ft.  in  closed  8  inch  test  section. 

-Mach  6  High  Reynolds  Number  Facility  (M6).  This  blow  down  facility  simulates  flight  at 
Mach  6  over  an  altitude  range  from  30,000  ft.  to  130,000  ft.  and  test  flow  temperatures  to  1000 
degrees  R  in  an  open  jet  test  section  with  a  13  inch  diameter  exit. 

-Twenty  Inch  Hypersonic  Wind  Tunnel.  This  blow  down  facility  simulates  flight  at  Mach  12  and 
Mach  14  over  an  altitude  range  from  120,000  ft.  to  150,000  ft.  and  test  flow  temperatures  to  2000 
degrees  R  in  an  open  jet  test  section  with  a  20  inch  exit  diameter. 

Contact:  WLMME,  DSN:  785-2139. 

A3.4.2.  Structural  Test  Facilities:  (Government  facilities  and  facilities  owned  and  operated  by  aero¬ 
space  contractors) 

A3.4.2.1.  Wright  Lab  -  Structures  Test  Facility.  Large-scale  combined  elevated  temperature  static 
and  dynamic  testing  of  aerospace  vehicles.  Contact:  WL/FIBT,  DSN:  785-5059. 

A3.4.3.  Aero-Thermodynamic  Heating  Indoor-Facilities: 

A3. 4. 3.1.  AEDC  -  Tunnels  B&C  for  aerothermal  testing.  Contact:  AEDC/DOFA,  DSN: 
340-7721. 

A3.4.3.2.  Wright  Lab  -  Radiant  heat  facility/convection  heat  facility.  Thermal  simulation  wing 
radiant  heat  -  55MW  maximum  power.  Contact:  WL/FIBT,  DSN:  785-5059. 

A3.4.4.  Birdstrike  Testing: 


122 


AFMAN99-110  3  JULY  1995 

A3.4.4.1.  AEDC  -  Birdstrike  Impact  Measurement  Facility.  Contact:  AEDC/DOFA,  DSN: 
340-5599. 

A3.4.5.  Component  Size  Environment/Climatic  Test  Chambers:  (also  see  Installed  System  Test 
Facilities  -  section  A3. 13.) 

A3.4.5.1.  Wright  Lab  -  Structures  Test  Environmental  Simulation/Fatigue  and  Fracture  Lab. 
Contact:  WL/FIBT,  DSN:  785-5059. 

A3.4.6.  Stability  and  Control  Testing: 

A3.4.6.1.  Wright  Lab  -  Flight  control  simulation  lab  (fixed,  motion,  and  in-flight)  at  Wright  Lab. 
Contact:  WL/HGD,  DSN:  785-4690. 

A3.4.7.  Combined  Environments/Acoustic  Test  Chamber: 

A3.4.7.1.  Wright  Lab  -  High  temperature,  high  intensity  acoustic  testing  facility.  Contact:  WL/ 
HBT,  DSN:  785-5059. 

A3.4.8.  Operations  and  Support  Facilities:  (detailed  deseriptions  are  contained  in  ALCs  Ground 
T«&E  Capability  Report,  dated  01  June  1994) 

A3.4.8.1.  OC-ALC  -  The  Engineering  Test  Support  Facility  supports  the  component  level 
hydraulics  pneumatics,  material,  vibration,  and  environmental  measurements.  Contact:  OC-ALC/ 
TIESS,  DSN:  336-5498, 

A3.4.8.2.  SA  -  ALC  -  The  Physical  Science  Lab  consists  of  five  (5)  laboratory  sections:  Chemical 
Science  Section,  Metallurgical  Science  Section,  Nondestructive  Inspection  and  Testing  Section, 
Process  Control  Section,  and  Quality  Verification  Section.  Contact:  SA-ALC/FMl,  DSN: 
945-4664. 

A3.4.8.3.  SM  -  ALC  -  The  Science  and  Engineering  Lab,  F-111  cold  proof  load  facility, 
non-destructive  inspection  facility,  Pneudraulic/Hydraulic  test  and  advanced  composites  facility 
have  the  capability  to  test  F-111,  A- 10,  F-1 17  and  other  airframe  components.  Contact:  SM  - 
ALC/TIEE,  DSN:  633-3147. 

A3.4.8.4.  WR  -  ALC  -  The  Corrosion  and  Environmental  Test  Facility,  Environmental  Test  Facil¬ 
ity,  and  Science  and  Engineering  Lab  have  the  capability  to  test  airframe  components  for  F-1 5, 
C-130,andC-141.  Contaet:  WR  -  ALC/TIECE,  DSN:  468-3238. 

A3.4.8.5.  00  -  ALC  -  The  Non-Destructive  Test  Facility  and  Science  and  Engineering  Lab  have 
the  capability  to  test  airframe  components  for  F-16  and  C-130s.  Contact:  OO  -  ALC/LIWBd, 
DSN:  458-4217. 

A3.5.  Propulsion: 

A3.5.1.  Uninstalled  Engine  Test  Cells: 

A3.5.1.1.  AEDC  -  The  Engine  Test  Facility  (ETF)  has  several  air-breathing  test  cells  capable  of 
simulating  flight  operational  conditions  over  a  wide  range  of  Mach  numbers  and  altitudes.  Mea¬ 
surements  can  be  made  under  precisely  controlled  conditions  required  to  determine  operational 
characteristics  of  major  subassemblies  (cascades,  fans,  and  compressors).  Contact:  AEDC/DOP, 
DSN:  340-5305. 
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A3.5.1.2.  Wright  Lab  -  Advanced  Turbine  Engine  and  Compressor  Research  Facility.  Contact: 
WL/POT,  DSN:  785-2331.  Combustion  Research  Facility:  Contact:  WL/POS/POP,  DSN: 
785-9991. 

A3.5.2.  Wind  Tlinnels: 

A3.5.2.1.  AEDC  -  Propulsion  wind  tunnels  16T  and  16S.  (See  Airframe  for  measuring  aerody¬ 
namic  performance  of  full-scale  engine  installation,  seetion  II.A.l.a.)  Contact:  AEDC/DOFA, 
DSN:  340-7721. 

A3.5.3.  Engine  Material  Properties  Labs: 

A3.5.3.1.  Wright  Labs  -  Evaluate  bearing  materials  and  lubricants  at  high  speed  and  high  temper¬ 
ature.  Contact:  WL/POSL,  DSN:  785-1286. 

A3.5.4.  Engine  Component  Test  Rigs: 

A3.5.4.L  AEDC  -  Engine  test  cells  may  be  modified  to  support  core  engine  and  combustor  test¬ 
ing.  Contact:  AEDC/DOP,  DSN:  340-5305. 

A3.5.4.2.  AEDC  -  Component  icing  and  de-icing  Research  and  Test  Facilities.  Contact:  AEDC/ 
DOP,  DSN:  340-5305. 

A3.5.5.  Engine  Operations  and  Support  Facilities: 

A3.5.5.1.  OC  -  ALC  -  The  First  Article  Testing  Unit,  Jet  Engine  Test  Facility  and  Cruise  Missile 
Engine  Test  Cells,  provide  the  eapability  for  turbine  engine  tests.  Contact:  OC  -  ALC/TIESS, 
DSN:  336-5498. 

A3.5.5.2.  SA  -  ALC  -  The  Engine  NDI  Facilities,  Cryogenic  SPIN  Test  Facility  and  Jet  Engine 
Test  Facility.  These  capabilities  are  dedicated  to  the  repair  and  production  of  fielded  engine  sys¬ 
tems.  T&E  usage  would  have  to  be  based  on  availability  and  feasibility.  Contact:  SA  -  ALC/ 
FM-1,  DSN:  945-4664. 

A3.6.  Avionics: 

A3.6.1.  Special  Access  Facilities: 

A3.6.1.1.  AFFTC  -  Other  special  test  capabilities  may  exist  that  can  be  applied  to  specific  pro¬ 
grams.  Contact  the  Single-Face-To-Customer  (SFTC)  office  with  your  requirements  so  an  assess¬ 
ment  can  be  made  on  special  test  facility  applicability.  Contact:  AFFTC/XPS,  DSN:  525-9250. 

A3.6.2.  RF  Antenna  Characterization  Facilities: 

A3.6.2. 1 .  Rome  Lab  -  Newport  Research  Facility  evaluates  antenna  systems  in  "farfield"  environ¬ 
ment.  Capability  to  pedestal  mount  F-16,  B-IB,  F-15,  etc.,  airframe  and  measure  radiation  pat¬ 
terns.  Contact:  RL/ERS,  DSN:  587-4217. 

A3. 6.2.2.  Rome  Lab  -  Stoekbridge  Research  Facility  evaluates  antenna  systems  and  ECM  threat 
responses  much  like  Newport  Researeh  Facility,  but  on  large  airframes  (C-130,  C-135,  B-52,  etc.). 
Contact:  RL/ERS,  DSN:  587-4217. 

A3. 6.2.3.  Rome  Lab  -  Preeision  Antenna  Measurement  System  (PAMS),  Verona  Research  Facil¬ 
ity,  Rome  Labs,  NY.  An  airborne  dynamic  radio  frequency  (RF)  antenna  measurement  facility 
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used  to  test  antenna  patterns  from  airborne  aircraft.  Ground  based  data  collection  receivers  gather 
in-flight  emission  data  and  analyze  data.  Contact;  RL/ERS,  DSN:  587-4217. 

A3.6.2.4.  AEDC  -  Focal  Plane  Characterization  Chamber,  characterization  by  flood  source  and 
point  source  for  focal  plane  arrays.  Contact:  AEDC/DOS,  DSN:  340-5599. 

A3.6.3.  Signature  Measurement  Facilities: 

A3.6.3.1.  AFDTC  -  Radar  Target  Scatter  (RATSCAT)  Facility,  at  Holloman  AFB  NM  provides 
the  capability  to  make  laboratory  quality  monostatic  and  bistatic  RCS  pole  measurements  [test 
article  weights  up  to  100,000  pounds  (lbs.)]  including  imaging.  RCS  measurements  range  from 
150  MHz  to  36  Giga  Hertz  (GHz)  using  co-  and  cross-polarization.  Additional  capabilities 
include  high  fidelity  RCS  model  fabrication,  and  RCS  prediction  and  analysis.  Contact:  46TG/ 
TGR,  DSN;  349-3365. 

A3.6.3.2.  AFDTC  -  RATSCAT  Advanced  Measurement  Facility  (RAMS),  at  Holloman  AFB  NM 
provides  similar  function  as  the  RATSCAT  Facility,  but  a  much  lower  RCS  levels  (very  low 
observable).  Contact:  46TG/TGR,  DSN:  349-3365. 

A3.6.3.3.  AEDC  -  SDIO/Plume  Data  Center  -  Engine  plume  measurements  in  tests  cells  and 
in-flight.  Contact:  AEDC/DOC,  DSN:  340-5599. 

A3.6.3.4.  Wright  Lab  -  Radar  Cross  Section  (RCS)  Measurement  Labs.  Contact;  WL/XPN, 
DSN:  785-5076. 

A3.6.3.5.  GO  -  ALC  -  The  ACM  Imaging  Radar  System  (AIRS)  is  used  to  diagnose  degradations 
of  the  cruise  missile  RCS.  Contact:  00  -  ALC/LIWGE,  DSN:  458-7351. 

A3.6.4.  Guidance  and  Navigation  Measurement  Facilities: 

A3.6.4.1.  AFDTC  -  Central  Inertial  Guidance  Test  Facility  (CIGTF)  at  HoUoman  AFB,  NM  per¬ 
forms  characterization  verification  of  navigation  and  guidance  components  and  systems  including 
jamming/spooling  vulnerability  testing.  Facilities  contain  the  largest  collection  of  precision  test 
tables  in  the  US  and  state-of-the-art  GPS  jamming/spooling  systems.  Conducts  the  full  gamut  of 
field  testing  including  aircraft,  ground  and  rocket  sled  testing.  Contact:  46TG/TGG,  DSN: 
349-2123. 

A3.6.5.  Weapons  Integration  Measurement  Facility: 

A3.6.5.1.  AFFTC  -  StoresAVeight/and  Inertia  System  (SWIS)  Facility,  at  Edwards  AFB  CA  pre¬ 
cisely  measures  store  weight  and  inertia  prior  to  aircraft  loading  (guided,  unguided,  and  missiles). 
Contact:  412TW/TSVW  ,  DSN:  525-9094. 

A3.6.5.2.  AEDC  -  Store  Carriage  Loads  and  Safety  of  Flight  Clearance  Tunnels  4T,  16T,  and 
Tunnel  A.  Contact:  AEDC/DOFA,  DSN:  340-7721. 

A3.6.5.3.  AFDTC  -  Fixed  and  mobile  facilities  for  measuring  store  mass  and  physical  properties. 
Contact:  AF  SEO,  Eglin  AFB,  DSN:  872-9052. 

A3 .6.5. 4.  AFDTC  -  Using  the  GWEF-PRIMES  Link,  parametrics  of  full-scale  munitions  in  the 
GWEF  connected  to  aircraft  in  the  PRIMES  can  be  measured  for  Captive-Carry,  midcourse,  and 
terminal  phases  of  weapon  system  employment.  Contact:  46TW/OGM,  Eglin  AFB,  DSN: 
872-4257. 

A3.6.6.  RF  Receiver/Processor  Measurement  Facilities: 
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A3.6.6.1.  Wright  Lab  -  The  RF  Receiver/Processor  Laboratory  has  developed  special  test  equip¬ 
ment  to  automate  receiver  measurements  using  common  test  criteria.  Contact:  WL/AAWP,  DSN: 
785-6131. 

A3. 6.6.2.  Wright  Lab  -  Dynamic/Combat  Electromagnetic  Environment  Simulator  (DEES/ 
CEESIM)  Facility  to  simulate  realistic  signal  environments  and  evaluate  receiver  performance. 
Contact:  WL/AAWA,  DSN:  785-4264. 

A3.6.7.  Electro-Optical  Measurement  Facilities: 

A3.6.7.1.  Wright  Lab  -  The  Electro-Optical  Receiver  Lab  is  a  development,  test  and  evaluation 
facihty  designed  to  perform  automatic  measurements  and  evaluations.  Most  optical  receiver  types 
have  been  tested  in  this  facility.  Contact:  WL/AAWP,  DSN:  785-2471. 

A3.6.7.2.  Wright  Lab  -  The  Optical  Research  Lab  provides  the  capability  for  test,  characterization 
and  development  of  novel  optical  components  and  devices  for  use  in  laser  radar  systems.  Unique 
capability  for  testing  and  evaluating  the  rapidly  maturing  technology  of  agile  beam  steering  using 
liquid  crystal  phased  arrays.  Contact:  WL/AARI-2,  DSN:  785-9614. 

A3.6.7.3.  Wright  Lab  -  The  Laser  Radar  Research  Lab  provides  analysis  and  testing  capability  for 
improved  electro-optical  (EO)  targeting  and  weapon  delivery  laser  radar  technologies.  Unique 
capability  or  2  micro  meter  Laser  Technology.  Contact  WL/AARI-2,  DSN  785-9614 

A3.6.8.  Communication  Avionics: 

A3.6.8.1.  Wright  Lab  -  The  Integrated  Electromagnetic  System  Simulator  (lESS)  provides 
real-time  dynamic  testing  of  advanced  integrated  communications,  navigation,  and  identification 
avionics  (ICNIA).  Contact:  WL/AAAI,  DSN:  785-2766. 

A3.6.9.  Radar  Measurement  Facilities: 

A3.6.9.1.  AFFTC  -  The  Integration  Faeility  For  Avionic  System  Testing  (IFAST)  provides  radar 
spread  benches  with  bays  on  side  of  building  to  allow  radar  clear  FOV  transmission/receive  (see 
HITL  Facilities).  Contact:  AFFTC/412TW/EWW,  DSN:  527-5404. 

A3.6.9.2.  AFFTC  -  A  Radar  Target  Generator  (RTG)  and  Dynamic  Angle  of  Arrival  (DAOA) 
Facility  is  currently  in  the  preliminary  design  phase  with  an  interim  capability  planned  for  GFY 
96.  This  will  allow  installed  airborne  radar  testing  over  controlled  ranges  of  operational  condi¬ 
tions.  This  part  of  the  Electronic  Combat  Integrated  Test  (ECIT)  Installed  System  Test  Facility 
(ISTF)  currently  under  development  (see  ISTF).  Contact:  AFFTC/412TW/EWD,  DSN: 
525-9426. 

A3.6.9.3.  AFFTC  -  Advanced  Radar  Test  Bed  (ARTB)  is  a  modified  C-141  capable  of  testing  var¬ 
ious  airborne  interceptor  radars  (see  Flying  Test  Beds).  Contact:  AFFTC/418TW/DOE,  DSN: 
527-4009. 

A3.6.9.4.  84th  TS  -  The  Radar  Test  Facility  (RTF)  can  provide  DT&E  capability  for  Air  Intercept 
(AI)  radars,  electronic  counter  measures  (ECM),  and  electronic  counter  countermeasures  (ECCM) 
in  a  controlled  repeatable  test  environment  [see  Hardware-In-The-Loop  (HITL)].  Contact:  84TS/ 
DOR,  DSN:  523-3376. 

A3.6.9.5.  Rome  Lab  -  Newport  Antenna  Range  for  testing  advanced  surveillance  radars.  Contact: 
RL/ERS,  DSN:  587-4217. 
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A3.6.10.  Avionics  Operations  and  Support  Facilities: 

A3.6.10.1.  OC  -  ALC  -  The  Avionics  Reliability  Center  has  the  ability  to  test  aircraft  inertial  nav¬ 
igation  systems,  altitude  heading  reference  systems  and  automatic  flight  control  systems.  Contact: 
OC  -  ALC/LHRU,  DSN:  336-5064. 

A3.6.10.2.  00  -  ALC  -  The  Nose  Radome  and  Radar  Antenna  Test  Facilities,  Avionics/Elec¬ 
tronic  Support  Facilities,  and  Optic  and  Photonic  System  Facilities  offer  T&E  capabilities  for  F-16 
and  other  platforms.  Contact:  00  -  ALC/LIRP,  DSN:  458-8080. 

A3. 6. 10.3.  SM  -  ALC  -  The  Electro-Optics  Test  Facility,  near  field  test  range,  in-flight  display 
systems  and  avionics  test  stands  offer  T&E  capabilities  for  F-111,  F-117,  and  A-10.  Contact: 
SM-ALC/TIEE,  DSN:  633-3147. 

A3.6.10.4.  WR  -  ALC  -  The  Indoor  Doppler  Antenna  Test  Range,  Microwave  Antenna  Test 
Facility,  and  Electronic  Failure  Analysis  Lab  provide  avionics  test  capabilities.  Contact:  WR  - 
ALC/LYPCA,  DSN:  468-4157. 

System  Integration  Laboratories  (SIL) 

These  facilities  integrate  hardware  and  software  components  up  to  the  subsystem  level  using  a 
table-top/spread-bench  environment.  Usually  a  specific  component  is  tested  in  a  SIL  with  the 
interface  components  simulated  (either  through  hardware  or  software).  These  facilities  usually 
reside  at  contractor  facilities  because  they  are  test  article  specific. 

A3.7.  Airframe: 

A3.7.1.  Aerodynamic/Flight  Control  Related  Labs:  (developing  leading-edge  technologies) 

A3.7.1.1.  AEDC  -  Tunnel  16T  fuU-scale  captive  flight  cruise  missile  testing,  functional  testing  of 
wing  deployment,  inlet  deployment,  engine  start  and  engine  operation.  Contact:  AEDC/DOFA, 
DSN:  340-7721. 

A3 .7. 1.2.  Wright  Lab  -  Real-Time  Piloted  engineering  Flight  Simulation  Lab  to  develop  flight 
vehicle  and  flight  control  technology.  Flight  Simulation  Lab  provides  multi  aircraft  full  mission 
level  (avionics,  controls,  weapon,  threat)  piloted  simulation  in  49  foot  dome,  20  foot  moving  base 
dome,  and  multiple  manned  control  stations  with  360  degrees  field-of-view  (FOV)  threat  and 
friendly.  Contact:  WL/FIGD,  DSN:  785-4690. 

A3.7.1.3.  Wright  Lab  -  Flight  Control  Activation  and  Hydraulics  Systems  Labs.  Contact:  WL/ 
HGS  DSN:  785-2831 

A3.7.2.  Materials  and  Structures  Labs:  (developing  leading-edge  technologies) 

A3.7.2.1.  AEDC  -  Aerothermal  Tunnel  C,  material  performance  in  interference  flows.  Contact: 
AEDC/DOS,  DSN:  340-5599. 

A3.7.2.2.  Wright  Lab  -  Aircraft  Structures  Test  Facility.  Contact:  WL/FIBT,  DSN:  785-5723. 

A3.7.3.  Vehicle  Subsystem  Labs:  (developing  leading-edge  technologies) 

A3 .7.3.1.  Wright  Lab  -  Landing  Gear  Development  Facility  to  perform  functional  and  qualifica¬ 
tion  test  on  landing  gear  assemblies  and  component  hardware.  Contact:  WL/FIVM,  DSN: 
785-2663. 
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A3.7.4.  Human  Factors/Cockpit  Integration  Related  Labs:  (developing  leading-edge  technolo¬ 
gies) 

A3.7.4.1.  Wright  Lab  -  Cockpit  and  Crew  Simulation  Lab.  Contact:  WL/FIP,  DSN:  785-2260. 
A3.7.4.2.  HSC  Labs,  Brooks  AFB,  TX.  Contact:  HSC/YA  DSN:  240-2345. 

A3.7.5.  Life  Support  Labs:  (developing  leading-edge  technologies) 

A3.7.5.1.  HSC,  Brooks  AFB,  TX.  Contact:  HSCA^A  DSN:  240-2345. 

A3.7.6,  Airframe  Operations  and  Support  Facilities:  (see  discussion  in  Measurement  Facilities) 

A3.8.  Propulsion: 

A3.8.1.  Propulsion  Integration  Facilities: 

A3. 8. 1.1.  AEDC  -  Engine  inlet  compatibility  at  ASTF  and  16T/16S  wind  tunnels.  Contact: 
AEDC/DOFA,  DSN:  340-7721.  ASTF  Contact:  AEDC/DOP,  DSN:  340-5305. 

A3. 8. 1.2.  AEDC  -  Engine  performance,  operability,  inlet  distortion  and  aeromechanical;  all  with 
airframe  interfacing  simulated  as  required  in  turbine  engine  altitude  cells.  Contact:  AEDC/DOP, 
DSN:  340-5305. 

A3. 8. 1.3.  Wright  Lab  -  Aero-Propulsion  Directorate.  Contact:  WL/POMX,  DSN:  785-2131. 

A3.8.2.  Engine  Operations  and  Support  Facilities:  (see  discussion  in  Measurement  Facilities, 
II.B.5.) 

A3.9.  Avionics: 

A3.9.1.  Avionics/Electronics  Related  Labs:  (for  developing  leading-edge  technologies) 

A3. 9. 1.1.  Wright  Lab  -  Avionics  Directorate.  Contact:  WL/AA,  DSN:  785-2620. 

A3.9.1.2.  Wright  Lab  -  Solid  State  Electronics  Directorate.  Contact:  WL/EL,  DSN:  785-2911. 

A3.9.1.3.  Wright  Lab  -  Integrated  Defensive  Avionics  Lab  (IDAL),  is  an  evolving  development/ 
evaluation  center  for  radar  warning  receiver  technology,  real-time  J-MASS  technology,  mission 
rehearsal  simulation  and  integrated  defensive  avionics.  Supports  both  open  and  closed  loop  sim¬ 
ulation  with  man/HITL  evaluations.  Contact:  WL/AAWA-2,  DSN:  785-4264. 

A3.9.2.  Facilities  used  for  Testing  Avionics/EC  Flight  Hardware/Software: 

A3.9.2.1.  AFFTC  -  Integration  Facility  for  Avionic  System  Testing  (IFAST).  This  facility  sup¬ 
ports  flight  test  programs  with  avionics  spread  benches  for  functional  verification,  anomaly  char¬ 
acterization,  integration  testing,  regression  testing,  software  evaluation,  and  aircrew 
familiarization  (including  weapons/munitions/armament  and  sensors).  Radar  bays  on  side  of 
building  allow  radar  clear  FOV  transmission/receiving.  Contact:  412TW/EWW,  DSN:  527-5404. 

A3.9.2.2.  WR  -  ALC  -  Electronic  Warfare  Avionics  Integration  Support  Facility  (EWAISF).  Pro¬ 
vide  engineering  and  platform  support  Air  Force  airborne  electronic  warfare  (EW)  systems  (ALQ, 
ALE,  and  ALR  series).  Contact:  WR  -ALC/LNEV,  DSN:  468-3691. 
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A3.9.2.3.  AFDTC  -  Pre-flight  Integration  of  Munitions  and  Electronics  Systems  (PRIMES)  Facil¬ 
ity  provides  a  subsystem/system  hot  bench  avionics  test  capability.  Contact:  46TW/TSWW 
DSN:  872-9354. 

A3.9.3.  Facilities  for  Testing  Weapons  Hardware/Software  Integration: 

A3.9.3.1.  AFFTC  -  Stores  Buildup/Integration  Facility  at  Edwards  AFB  CA.  DOD  explosive  cer¬ 
tified  facility  used  to  evaluate  weapon  integration  with  aircraft  stores  management  systems.  Has 
capability  to  evaluate  cruise  missile  integration.  Contact:  AFFTC/412  TW/TSVW,  DSN: 
525-9094. 

A3.9.4.  Avionics  Operations  and  Support  Facilities; 

A3.9.4.1.  OC  -  ALC  -  Avionics  Integrated  Support  Facilities  are  used  to  conduct  maintenance  and 
integration  testing  to  verify  and  analyze  software  problems,  to  design  code  and  test  changes  to  cor¬ 
rect  identified  problems  and  to  test  new  software  versions.  Avionics  Integration  Support  Facility 
(AISF)  facilities  include:  E-3  AISF;  E-4B  SIL;  B-52  AISF;  B-IB  AISF;  and  CM  AISF.  Contact: 
OC  -  ALC/TIESS,  DSN:  336-5498. 

A3.9.4.2.  OO  -  ALC  -  F-16  Software  Development  and  Engineering  Test  Station  and  Software 
Technology  Support  Center  supports  avionics  integrated  support  testing.  Contact:  OO  -  ALC/ 
LIWB4,  DSN:  458-4217. 

A3.9.4.3.  SA  -  ALC  -  C-17  Avionics  Integrated  Support  Facility(currently  under  construction). 
Contact:  SA  -  ALC/LC,  DSN:  945-3495. 

A3 .9.4.4.  SM  -  ALC  -  The  Advanced  Electronics  Center,  Extendible  Integration  Support  Facility 
(EISF),  System  Technology  Insertion  Facility,  F-111  AISF,  A-10  ISF,  and  the  Operational  Test 
Program  (OTP)  provides  software  integration  testing  capabilities.  Contact:  SM  -  ALC/TIEE, 
DSN:  633-3147. 

A3.9.4.5.  WR  -  ALC  -  Avionics  Integrated  Support  Facilities  include  the  following  capabilities: 
GPS  ISF;  C-130  Self  Contained  NAV  SYS  ISF;  Joint  Stars  ISF;  EW  AISF;  F-15  AISF;  Pave  Tack 
AISF;  and  SOF  AISF.  Contact:  WR-ALC/LNEV,  DSN:  468-3691. 

Hardware  In-The-Loop  (HITL)  Facilities 

These  facilities  provide  the  capability  to  test  actual  subsystem  level  hardware  in  a  closed-loop 
indoor  lab  environment  (for  example,  iron-bird  labs  and  fuel  system  labs).  Generally,  they  are 
more  sophisticated  than  SILs  and  typically  reside  in  contractor  facilities.  Subsystems  and  sensors 
other  than  the  one  under  test  are  simulated.  A  government  example  is  the  Integration  Facility  For 
Avionic  System  Testing  (IFAST)  located  at  the  Air  Force  Flight  Test  Center,  Edwards  AFB  CA. 

A3.10.  Airframe: 

A3.10.1.  Subsystems  Test  Facilities: 

A3.10.1.1.  Wright  Lab  -  Tire  and  Brake  Dynamometers.  Contact:  WL/DOR,  DSN:  785-4404. 

A3. 10. 1.2.  Wright  Lab  -  Hydraulics  System  Simulator.  Contact:  WL/DOR,  DSN:  785-4404. 

A3. 10.1.3.  Wright  Lab  -  Structures  Test  Facility.  Contact:  WL/FIBT,  DSN:  785-5059. 

A3. 10. 1.4.  AFDTC  -  Climatic  Laboratory  for  Airframe  Testing  .  Contact:  46TW/TSWL,  DSN: 
872-4610. 
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A3. 10.1.5.  Wright  Lab  -  Vibroacoustics  Test  Facilities  .  Contact:  WL/FIBG,  DSN:  785-2544 

A3. 10.2.  Operations  and  Support  Facilities  (see  ALC  capabilities  in  Measurement  and  SEL  Testing 
Section) 

A3.11.  Propulsion; 

A3.11.1.  Uninstalled  Engine  Sea  -Level  T&E: 

A3.11.1.1.  AEDC  -  Sea  Level  Test  Stand  (SLT-1).  Contact:  AEDC/DOP,  DSN:  340-5305. 

A3. 11. 1.2.  AEDC  -  APTU,  Tj ,  up  to  2000  degrees  R.  Contact:  AEDC/DOF,  DSN:  340-7721. 

A3.11.2.  Uninstalled  Engine  Altitude  T&E: 

A3. 11.2.1.  AEDC  -  Engine  Test  Facility  (ETF),  and  Aeropropulsion  Systems  Test  Facility 
(ASTF)  are  used  for  the  evaluation  of  air-breathing  engine  performance,  engine/inlet  dynamics, 
engine  operability  transients,  engine  aeromechanical  behavior,  engine  mission  simulations, 
engine/inlet  and  eomponents/missile  mission  simulation,  engine  durability  [Ram  Accelerated 
Mission  Testing(AMT)],  and  nozzle  vectoring  and  development.  Contact:  AEDC/DOP  DSN: 
340-5305 

A3.11.3.  High  AOA/Sideslip  T&E: 

A3. 11. 3.1.  AEDC  -  Aeropropulsion  Systems  Test  Facility  (ASTF),  provides  one  test  cell  with 
“free  jet”  eapability  to  perform  altitude  testing  for  performance  and  operability  with  full-up  inlet/ 
engine  combination  at  varieties  of  sideslips  and  angles-of-attack.  Contact:  AEDC/DOP  DSN: 
340-5305. 

A3.11.3.2.  AEDC  -  Propulsion  Wind  Tunnels  16T  and  16S  engine  inlet  compatibility.  Contact: 
AEDC/DOFA,  DSN:  340-7721. 

A3.11.4.  Uninstalled  Engine  Icing/de-icing/anti-icing  QualiHcation  T&E: 

A3. 11. 4.1.  AEDC  Engine  Test  Facility  (ETF)  and  Aeropropulsion  System  Test  Facility  (ASTF) 
provide  test  cells  with  the  capability  to  perform  inlet/engine  or  uninstalled  engine  anti-icing/ 
de-icing  qualification  testing  at  various  altitude/Mach  No.  and  rain  cloud  conditions.  Contact: 
AEDC/DOP  DSN:  340-5305. 

A3.11.5.  Climatic  Testing  of  Engine/Components: 

A3. 11. 5.1.  McKinley  Climatic  Laboratory  Engine  Test  Cells,  provides  the  capability  to  test  func¬ 
tioning  engines  in  extreme  temperature  conditions.  Contact:  Eglin  AFB,  DSN:  872-4601. 

A3. 11. 5.2.  AEDC  Engine  Test  Facility  (ETF)  and  Aeropropulsion  Test  Facility  (ASTF)  provide 
test  cells  with  the  capability  of  thermally  soaking  and  operationally  testing  of  engines  at  extreme 
temperatures  ranging  from -lOOxF  to +1080xF.  Contact:  AEDC/DOP  DSN:  340-5305. 

A3.11.6.  Operations  and  Support  Facilities:  (see  ALC  capabilities  in  Measurement  and  SIL  testing 
section) 

A3.12.  Avionics: 

A3.12.1.  Navigation: 


130 


AFMAN99-110  3  JULY  1995 

A3. 12. 1.1.  AFDTC  -  Central  Inertial  Guidance  Test  Facility  (CIGTF).  Contact:  746th  Test 
Squadron,  DSN:  349-2123. 

A3.12.2.  Facilities  used  for  Testing  Avionics  and  EC  Integration  Flight  Hardware/  Software: 

A3. 12.2.1.  AFFTC  -  Integration  Facility  for  Avionic  Support  Testing  (IFAST).  This  facility  sup¬ 
ports  flight  test  programs  with  avionics  spread  benches  for  functional  verification,  anomaly  char¬ 
acterization,  integration  testing,  software  evaluation,  and  aircrew  familiarization.  Radar  bays  on 
side  of  building  allow  radar  clear  FOV  transmission/receiving.  Contact:  FFTC/412TW/EWW, 
DSN:  527-5404. 

A3. 12.2.2.  Air  Force  Electronic  Warfare  Evaluation  Simulator  (AFEWES).  Provides  high  den¬ 
sity  signal  environment  to  test  ECM  techniques  using  manned  threat  simulators.  Can  test  compo¬ 
nents  and  subsystems  in  either  open  or  closed  loop  mode.  Contact:  46  TW/TSWW,  DSN: 
872-3410. 

A3. 12.2.3.  Real-time  Electromagnetic  Digitally  Controlled  Analyzer  Processor  (REDCAP).  Pro¬ 
vides  real-time  man-in-the-loop  hybrid  simulation  of  Integrated  Air  Defense  Systems  (IADS)  to 
test  component  performance  and  simulated  aircraft  survivability.  Contact:  46TW/TSWG  DSN: 
872-3410. 

A3.12.2.4.  AFDTC -PRIMES  Facility  at  EglinAFB.  Contact:  AFDTC/DRC  DSN:  872-9650 

A3.12.3.  Weapons  Integration  HITL  Facilities: 

A3. 12.3.1.  AFFTC  -  Stores  Buildup/Integration  Facility.  DOD  explosive  certified  facility  used  to 
evaluate  weapon  integration  with  aircraft  stores  management  systems.  Has  capability  to  evaluate 
cruise  missile  integration.  Contact:  412  TW/TSVW  DSN:  525-9094. 

A3. 12.3.2.  AFDTC  -  Guided  Weapons  Evaluation  Facility  (GWEF).  Provides  weapon  system 
and  subsystem  T&E  using  digital,  HITL,  and  ECM  simulation.  Using  the  GWEF-PRIMES  link, 
HITL  testing  of  full  scale  munitions  in  the  GWEF,  connected  to  aircraft  in  the  PRIMES,  is  pro¬ 
vided  for  captive-carry,  midcourse,  and  terminal  phases  of  weapon  system  employment.  Contact: 
46  TW/TSWQ  DSN:  872-9988. 

A3. 12.3.3.  AEDC  -  IR  Guided  Weapons,  Direct  Write  Scene  Generator  Facility.  Contact:  AEDC/ 
DOC,  DSN:  340-5599. 

A3.12.4.  Avionics/Electronics  Labs  for  HITL  Testing: 

A3. 12.4.1.  Wright  Lab  -  The  Dynamic  Infrared  Missile  Evaluator  (DIME)  facihty  provides  test¬ 
ing  capability  in  the  areas  of  IR  guided  weapon  exploitation  and  infrared  countermeasures 
(IRCM)  technique  development.  Provides  a  range  of  capabilities  from  aircraft  signature  studies  in 
the  visible  and  IR  portion  of  the  spectrum  to  real-time  HITL  flyout  simulation.  EO/IRCM  tech¬ 
niques  can  be  developed  using  both  digital  modeling  and  hardware  tests.  Contact:  WL/AAWW, 
DSN:  785-4174. 

A3. 12.4.2.  Wright  Lab  -  The  Optical  Collimator  Facility  provides  research,  test  and  analysis 
capability  for  tactical  and  strategic  EO  and  laser  radar  systems  under  simulated  environmental 
conditions.  Contains  a  100  inch  diameter  optical  collimating  mirror  housed  in  a  vacuum  chamber 
which  can  be  evacuated  to  simulate  a  270,000  ft  altitude.  Contact:  WL/AARI-2,  DSN:  785-9614. 
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A3. 12.4.3.  Wright  Lab  -  Infrared  Laboratory  capable  of  characterizing  and  supporting  the  devel¬ 
opment  of  EO  imaging  sensor  systems.  Contact:  WL/AARI-2,  DSN:  785-9614. 

A3. 12.4.4.  Wright  Lab  -  Air  Force  Sensor  Evaluation  Center  (AFSEC)  provides  capability  for 
T&E  and  development  imaging  sensor  system.  Capable  of  emulating  actual  flight  environments 
with  equipment  installed  in  aircraft  section.  Contact:  WL/AARF,  DSN:  785-5406. 

A3. 12.4.5.  Wright  Lab  -  Extremely  broad  bandwidth  anechoic  facility  to  perform  exploitation  of 
foreign  missile  system  and  develop  countermeasures  against  the  missiles.  Contact:  WL/ 
AAWW-2,  DSN:  785-6504. 

A3.12.5.  Operations  and  Support  Facilities:  (see  ALC  capabilities  in  the  Measurement  and  SIL 
testing  section) 

A3.12.6.  Radar  Test  Facilities: 

A3. 12.6.1.  AFFTC  -  The  IFAST  Radar  Test  Station  (RTS)  provides  a  radar  ground  test  facility 
that  is  configurable  to  several  different  avionics  configurations.  The  RTS  can  operate  the  radar 
using  airborne  targets  or  simulated  targets  via  a  signal  generator.  The  F-15  avionics  laboratory 
also  located  in  the  IFAST  provides  a  core  avionics  and  radar  ground  test  facility  reconfigurable  to 
support  avionics  flight  tests.  Contact:  412TW/EWW  DSN:  527-5404. 

A3. 12.6.2.  84th  Test  Squadron  -  The  Radar  Test  Facility  (RTF)  conducts  operational  test  and 
evaluation  (OT&E)  and  developmental  test  and  evaluation  (DT&E)  on  airborne  interceptor  radars, 
electroinc  countermeasures  (ECM),  and  electronic  counter-countermeasures  (ECCM).  The  RTF 
provides  F-15,  F-16,  and  AIM-7F/M/H  integrated  weapon  system  testing  in  a  controlled,  repeat- 
able  test  environment.  Contact:  84TS/DOR,  DSN:  523-3376. 

Installed  System  Test  Facilities  (ISTF) 

These  facilities  provide  the  capability  to  evaluate  systems  which  are  installed  on  the  host  platform/ 
airframe  while  residing  within  a  controlled  indoor  environment.  Examples  include,  anechoic 
chambers,  limit  cycle/ground  resonance  test  stands,  structural  loads  stands  and  full-scale  wind 
tunnel  tests. 

A3.13.  Airframe: 

A3.13.1.  Ground  Resonance/Vibration/Limit  Cycle  Test  (GRT/GVT)  Systems.  Predominantly 
conducted  in  aerospace  contractor  owned  and  operated  facilities 

A3. 13. 1.1.  AFFTC  -  Limit  Cycle/GRT  system.  Contact:  AFFTC/XPS,  DSN:  525-9266. 

A3. 13. 1.2.  Wright  Lab  -  Ground  Vibration  Tests.  Contact:  WL/FTBG  DSN:  785-5200 

A3.13.2.  Egress/Ejection  Systems  T&E:  IHigh  Speed  Test  Track.  Contact:  AFDTC/Holloman, 
DSN:  349-2133. 

A3.13.3.  Vulnerability/Live  Fire  T&E: 

A3. 13. 3.1.  Wright  Labs  -  Live  Fire  capability  up  to  30mm  rounds.  Contact:  WL/FIV,  DSN: 
785-4840. 

A3.13.3.2.  High  Speed  Test  Track.  Contact:  AFDTC/Holloman,  DSN:  349-2133. 

A3.13.4.  MI/EMC/Lightning  Effects  T&E: 
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A3.13.5.  Climatic  Testing:  (see  A3.4.4.  for  component  testing) 

A3. 13.5.1.  McKinley  Climatic  Lab,  AFDTC,  Eglin  AFB,  FL.  Info  (size,  temp  ranges,  etc.).  Con¬ 
tact:  AFDTC/Eglin,  DSN:  872-4601. 

A3.13.6.  Structural  Load  T&E:  (see  Measurement  Facilities) 

A3.13.7.  ECM/ECCM  Testing.  High  speed  test  track.  Contact:  846th  Test  Squadron,  DSN: 
349-2133 

A3. 14.  Propulsion: 

A3.14.1.  Installed  Engine  Test  Facilities: 

A3. 14. 1.1.  AFFTC  -  Horizontal  Thrust  Stand.  Provides  direct  measurement  of  thrust  at  ground 
level  (approx.  2200  ft  altitude)  at  ambient  conditions.  Can  accommodate  1  Mlb  aircraft  and  asym¬ 
metric  thrust.  Contact:  AFFTC/DOEF,  DSN:  525-9017. 

A3.14.2.  Climatic  T&E:  (see  Airframe  Installed  System  Test  Facilities,  A3.13.) 

A3.14.3.  Live  FireA^ulnerability  T&E:  (see  Airframe  Installed  System  Test  Facilities,  A3.13.) 

A3.15.  Avionics: 

A3.15.1.  EC  and  Avionics  Integration: 

A3. 15. 1.1.  AFFTC  -  Benefield  Anechoic  Chamber  (BAF)  412TW/EWWA,  Edwards  AFB,  CA 
Supports  flight  and  ground  test  programs  with  a  large  (264x250x70  ft.)  anechoic  chamber,  the 
largest  in  the  world.  Attenuation/isolation  ranges  from  120  to  200  dB  within  the  chamber’s  RF 
quiet  zone.  A  turntable  and  hoist  located  near  the  center  of  the  chamber  allows  testing  to  be 
accomplished  at  multiple  azimuths  and  elevations  using  free-space  RF  techniques.  Signal  gener¬ 
ation  system  provides  an  extensive  emitter  library  capable  of  simulating  land-based,  sea-based, 
air-to-air  red,  blue,  and  gray  systems  primarily  with  the  frequency  range  of  0.5  to  18.0  GHz.  Test 
environments  are  possible  well  outside  this  frequency  range.  Contact:  412TW/EWWA,  DSN: 
527-0840. 

A3. 15. 1.2.  AFFTC  -  Electronic  Combat  Integrated  Test  (ECIT)  Development  Program.  The 
ECIT  Program  will  provide  the  Avionics  Test  and  Integration  Complex  (ATIC)  with  a  secure 
ground  test  environment  to  test  and  evaluate  current  avionics  systems  and  future  complex, 
highly-integrated  software,  intensive  avionics  suites  installed  on  the  combat  aircraft.  Avionics 
suites  include  radar,  electronic  warfare,  communications,  navigation  and  identification  (CNI), 
Infrared  (IR),  weapon  management,  sensor  management  and  other  functions.  The  ECIT  will  cre¬ 
ate  realistic  multispectral,  multiple  player,  and  synthetic  environments  so  that  integrated  avionics 
can  be  tested.  The  initial  development  phase,  infrastructure  and  generic  test  capability  (I&GTC) 
is  planned  for  development  start  in  GFY95.  Contact:  412TW/EWD  DSN:  525-9426. 

A3.15.2.  Weapons  Integration: 

A3. 15. 2.1.  46TW/TSWW  -  Pre-flight  Integration  of  Munitions  and  Electronics  Systems 
(PRIMES).  Provides  an  RF  environment  in  108x78x30  ft.  anechoic  chamber  to  stimulate  the 
weapon/aircraft.  The  facility  has  a  0.5  to  18  GHz  dynamic  threat  signal  simulator,  as  well  a  sub¬ 
system/system  hot  bench  test  room  and  data  analysis  package.  Using  the  GWEF-PRIMES  Link, 
integrated  performance  testing  of  full  scale  munitions  in  the  GWEF,  connected  to  aircraft  in  the 
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PRIMES,  is  provided  for  captive-carry,  midcourse  and  terminal  phases  of  weapon  system  employ¬ 
ment.  Contact:  46  TW/TSWW,  DSN:  872-9354. 

A3.15.3.  EMI/EMC,  Climatic,  and  Vulnerability  T&E:  (see  Airframe  A3.13.) 

A3. 15.3.1.  AFDTC  -  PRIMES  EMI/EMC  capability.  Contact:  46TW/rSWW  DSN:  872-9354. 

A3.15.4.  Threat  Exploitation  Facilities: 

A3. 15.4. 1 .  Wright  Lab  -  Hangar  4  anechoic  facility  capable  of  performing  threat  radar  and  missile 
technical  exploitation.  Allows  development  of  ECM  techniques  against  actual  threat  hardware 
and  test  of  operational  ECM  systems  under  controlled  conditions.  Exacts  data  from  damaged 
components  and  complex  printed  circuit  boards.  Contact:  WL/AAWP-2,  DSN:  785-2471. 

Open  Air  Ranges  (OAR) 

This  category  of  T&E  resources  is  divided  into  two  groups,  controlled/instrumented  ranges  and 
real-world  environments.  Although,  the  weapon  system  should  be  tested  in  a  controlled  environ¬ 
ment,  valuable  data  may  be  gathered  from  real-world  scenarios  which  may  effect  system 
upgrades/modifications. 

A3.16.  Airframe: 

A3.16.1.  Basic  Airworthiness:  (performance  and  flying  quahties) 

A3. 16. 1.1.  AFFTC  -  Besides  fully  instrumented  ranges  [with  global  positioning  system  (GPS), 
visual,  or  radar  tracking]  for  conducting  Airworthiness  T&E  (flying  qualities,  performance,  high 
AOA,  structural  tests,  etc.),  the  AFFTC  has  unique  airworthiness  related  facilities  for:  air  data  cal¬ 
ibration  (tower  and  pacer),  aircraft  weight  and  balance  (up  to  IM  lbs),  gun  harmonization  ( 30mm 
gun  butt),  large  lakebed  (multiple  emergency  landing  options),  instrumentation  lab  (Class  n  capa¬ 
ble).  Contact:  412  TW/TSF,  DSN:  525-9089. 

A3. 16. 1.2.  System  Effectiveness  Data  System  (SEDS)  -  Used  to  store  and  process  failure  and 
maintenance.  Generates  reliability  and  maintainability  parameters  during  a  development  test  pro¬ 
gram.  Contact:  412  TW/TSSR  DSN:  525-9128 

A3.16.2.  TF/TA  Development: 

A3. 16.2.1.  AFFTC  -  Haystack  Butte  Route.  This  is  a  5  X  26  mi.  fuUy  instrumented  and  surveyed 
route  (to  2  ft  elevation  resolution).  It  contains  a  smooth  gradual  incline  over  desert  terrain  with  an 
abrupt  terrain  change  at  Haystack  Butte  for  fly-up  command  checkout  and  is  ideal  for  initial  TF/ 
TA  development.  Contact:  412  TW/TSR,  DSN:  525-3050. 

A3. 16.2.2.  AFFTC  -  R-2508  airspace  terrain  following/terrain  avoidance  (TF/TA)  training  routes. 
Multiple  routes  over  varying  terrain  (desert  and  mountainous)  with  sited  navigation  waypoints. 
Contact:  412  TW/TSR,  DSN:  525-3050. 

A3. 16.2.3.  AFFTC  -  Utah  Test  and  Training  Range  (UTTR)  Hill  AFB  UT.  Land  areas  beneath 
the  17,000  square  nautical  mile  UTTR  airspace  contains  some  30  different  DMA  Terrain  Contour 
Matching  (TERCOM)  maps  in  all  four  categories  of  Landfall,  Midcourse,  Enroute,  and  Terminal. 
Additionally,  all  categories  of  land  surface  roughness  are  available  from  Very  Smooth  to  Very 
Rough.  Airspace  vertical  limits  allow  aircraft,  cruise  missile,  or  UAV  TF/TA  flight  operations 
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down  to  100  feet  AGL  throughout  the  range,  and  down  to  the  surface  over  DOD  owned  land  areas. 
Contact:  545TG/XRP,  DSN:  458-7852. 

A3.16.3.  Aerial  Delivery: 

A3. 16.3.1.  AFFTC  -  In  addition  to  in-flight  aerial  delivery  T&E  (AFFTC  is  DOD  center  of 
expertise  for  aerial  delivery)  including  low  altitude  parachute  extraction  system  (LAPES)  (lake- 
bed  is  ideal  for  initial  LAPES  T&E),  the  AFFTC  has  facilities  which  support  aerial  delivery  (para¬ 
chute  drying  tower,  parachute  rigging  facility,  load  buildup  facility,  and  fully  instrumented  drop 
ranges).  Contact:  417  TS/EN,  DSN:  527-6404. 

A3.16.4.  Structural  Flight  Test:  (loads,  aeroelasticity,  weight  and  balance,  and  vibroacoustics) 

A3. 16.4.1.  AFFTC  -  Real-time  (near)  system  is  available  in  mission  control  rooms  to  provide 
structural  damping,  frequency,  and  structural  response  of  air  vehicles.  Contact:  412TW/TSFS 
DSN:  525-9091. 

A3.16.5.  Initial  Weapons  Separation  T&E:  (also  see  A3.18.) 

A3. 16.5.1.  AFFTC  -  Precision  Impact  Range  Area  (PIRA)  A  75  sq.  mi.  precision  instrumented 
Class-B  range  (with  supersonic  capability)  for  overland  dropping  inert  stores.  For  weapon  system 
development,  the  range  is  used  for  weapon  separation  T&E  (less  than  1  ft  resolution)  and  prelimi¬ 
nary  weapons  integration  T&E.  Contact:  412  TW/TSR,  DSN:  525-3050. 

A3. 16.5.2.  AFDTC,  Eglin  AFB  has  capability  for  weapon  separation  T&E  -  Contact:  460G/ 
OGMD  DSN:  872-9551. 

A3. 16.5.3.  Weapon/Avionics  Integration  T&E  (see  A3.18.1.). 

A3.16.6.  Air  Refueling  Certification  and  Development: 

A3. 16.6.1.  AFFTC  -  Instrumented  KC-135  and  KC-10.  Contact:  412TW/TSSS,  DSN:  525-9090. 

A3.16.7.  In-flight  Icing  and  Rain: 

A3. 16.7.1.  AFFTC  -  Instrumented  KC-135.  The  aircraft  uses  bleed  air  and  water  nozzles  to  con¬ 
trol  droplet  size,  density,  etc.  for  producing  artificial  icing  cloud  (4-6  ft  diameter).  Contact: 
412TW/TSSS,  DSN:  525-9090. 

A3.16.7.2.  AFFTC  -  Instrumented  CH-53.  Contact:  US  Army  STEAT/AQ  CE,  DSN:  527-3901. 

A3.16.8.  Aircraft  Braking  and  Mechanical  Systems  T&E: 

A3. 16. 8.1.  AFFTC  -  Aircraft  braking  performance  is  evaluated.  15,000  ft  runway  plus  lakebed 
overrun  minimize  risks.  Contact:  41 2TW/TSSS  DSN:  525-9090. 

A3. 16.8.2.  AFFTC  -  Mechanical  Systems  T&E  (deceleration  devices).  Contact:  412TW/TSVI, 
DSN:  525-4820. 

A3.16.9.  Unmanned  Air  Vehicle  (UAV)  Flight  Test: 

A3. 16.9.1.  AFFTC  -  Utah  Test  and  Training  Range  (UTTR)  Hill  AFB  UT.  A  17,000  square  nau¬ 
tical  mile  airspace,  overland,  test  range  consisting  of  Restricted  Areas  and  MO  As  overlying  DOD 
and  Bureau  of  Land  Management  (BLM)  government  lands.  Has  UAV  launch  and  recovery 
areas/facilities,  including  13,000  foot  runway  adjacent  to  administrative  and  maintenance  support 
facilities  at  Dugway  Proving  Ground.  Complete  range  instrumentation  services  are  provided  for 
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TSPI,  TM,  data  processing  and  display,  including  remote  command/control  and  range  safety  flight 
termination  capabilities.  Contact:  545TG/XRP,  DSN:  458-7852. 

A3.16.10.  Cruise  Missile  (CM)  Flight  Test: 

A3. 16.10. 1 .  AFFTC  -  Utah  Test  and  Training  Range  (UTTR)  Hill  AFB  UT.  A  17,000  square  nau¬ 
tical  mile  airspace,  overland,  test  range  with  all  variations  of  terrain  (very  smooth  to  very  rough) 
and  categories  of  TERCOM  maps  for  TF/TA  cruise  missile  operations.  All  test  terrains  and  maps 
are  contained  within  the  range  boundaries.  Range  has  complete  TSPI,  TM,  and  command/control/ 
termination  capabilities  and  airburst  or  ground  impact  scoring.  An  instrumented  TF  route/corridor 
(IR-200)  links  the  UTTR  with  Nellis-Tonapah,  Edwards-China  Lake,  and  Vandenburg-Pt.  Mugu 
Ranges  for  testing  realistic  range/endurance  and  navigational  accuracy.  Contact:  AFFTC 
412TW/TSR,  DSN:  525-3050,  or  Contact:  UTTR  545TG/XRP,  DSN:  458-7852. 

A3. 16. 10.2.  AFDTC  -  Eglin  AFB  has  a  Cruise  Missile  Test  Capability.  Contact:  46TW/TSWW 
DSN:  872-9354. 

A3.17.  Propulsion: 

A3.17.1.  Propulsion  Flight  Test.  Instrumentation  telemetry  available  in  real-time 

A3. 17. 1.1.  AFFTC  -  20,000  sq.  mi.  of  airspace,  with  GPS,  radar,  optical  tracking  capability. 
Real-time  and/or  post-flight  data  TM  available  using  Ridley  Mission  Control  Center  (RMCC). 
Contact:  412  TW/TSR,  DSN:  525-3050. 

A3.17.1.2.  ALCs  (WR,  00,  and  SM)  -  Contact  Appropriate  Center 

A3.17.2.  Propulsion  Flight  Test.  Hazardous  propulsion  T&E  (new  single-engine  aircraft  T&E,  high 
AOA,  very  high  speed  (hypersonic,  etc.) 

A3. 17.2. 1 .  AFFTC  -  Provides  numerous  emergency  landing  options  on  5x12  mile  lakebed  and/or 
15,000  ft  runway.  Contact:  412  TW/TSR  DSN:  525-3050. 

A3.17.3.  UAV  and  Cruise  Missile  Propulsion-related  T&E:  (see  A3.16.) 

A3.18.  Avionics: 

A3.18.1.  Weapons  Integration:  (weapon/stores  separation  T&E  is  in  A3.16.5.) 

A3. 18. 1.1.  AFFTC  -  The  Dual  Air-To-Ground  Range  (DAGRAG)  is  a  continual  low  altitude 
air-to-surface  gunnery,  bombing,  and  rocket  range.  Contact:  412TW/DOEA,  DSN:  525-9060. 

A3. 18. 1.2.  UTTR  for  very  large  footprint  weapons.  Contact:  AFFTC/XPS,  DSN:  525-9266. 

A3.18.2.  EC  Integration/Survivability  T&E: 

A3. 18.2.1.  AFFTC/EW  Directorate  -  EW  Integration  and  Effectiveness  Hight  Testing  at  specific 
ranges.  Contact  412TW/EW  DSN:  525-7772. 

A3. 18.2.2.  AFDTC/EW  SFTC  Office  -  EW  Integration  and  Effectiveness  Flight  Testing  at  spe¬ 
cific  ranges.  Contact:  AFDTC/DRC  DSN:  872-9650. 

A3.18.3.  Special  Access  Facilities: 
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A3. 18.3.1.  AFFTC  -  Other  special  test  capabilities  may  exist  that  can  be  applied  to  specific  pro¬ 
grams.  Contact  the  SFTC  office  with  your  requirements  so  an  assessment  can  be  made  on  special 
test  facility  applicability.  Contact:  AFFTC/XPS,  DSN:  525-9250. 

A3.18.4.  Cruise  Missile  Flight  Test;  (see  A3.16.10.) 

A3.18.5.  Laboratory  Open  Ranges: 

A3. 18.5.1.  Wright  Lab  -  The  Laser  Communications  Lab  (LCL)  provides  a  capability  for  devel¬ 
opment  and  test  of  next  generation  Laser  and  RF  communications  systems.  Provides  hne-of-sight 
access  to  remote  locations  for  propagation  and  data  collection/recording.  Contact:  WL/AAAI-2, 
DSN:  785-3455. 

A3. 18. 5.2.  Wright  Lab  -  Laser  IRCM  Development  (LID)  range  for  short  range,  closed  loop 
IRCM  development  testing.  Provides  robust  measurement  of  IR  or  effectiveness.  Contact:  WL/ 
AAWD,  DSN:  785-3489. 

A3.18.5.3.  Wright  Lab  -  EO  sensor  evaluation  range  is  capable  of  evaluation  EO  sensors,  obtain¬ 
ing  target  and  background  signatures,  and  determining  effect  of  camouflage  on  sensor  perfor¬ 
mance.  Contact:  WL/AARI,  DSN:  785-9609. 

Flying  Test  Beds 

A3.19.  Airframe: 

A3.19.1.  In-flight  Simulators: 

A3. 19. 1.1.  Wright  Lab  -  Total  In-flight  Simulator  (TIFS)  -  A  modified  NC-131H  aircraft  owned 
by  Wright  Labs  and  operated  by  Calspan  Corp.  which  is  configured  with  simulation  cockpit  (can 
be  customized  to  match  test  aircraft)  in  addition  to  flight  cockpit.  Onboard  digital  computers 
allow  6  degree-of-freedom  rigid  body  simulation  of  an  aircraft’s  control  system/flying  qualities. 
In-flight  simulation  give  pilots  cues  not  available  in  ground  based  simulations.  Contact:  WL/ 
FIGD,  DSN:  785-3853. 

A3. 19. 1 .2.  Wright  Lab  -  Variable  Stability  In-flight  Simulator  Test  Aircraft  (VISTA).  A  modified 
F-16D  Block  30,  Peace  Marble  owned  by  AF  and  operated  by  Calspan  Corp.  The  onboard  digital 
computers  make  the  aircraft  capable  of  simulating  the  flying  characteristics  of  high  performance 
aircraft.  Contact:  WL/FIGD,  DSN:  785-3853. 

A3.19.2.  Captive  Carriage  Platforms: 

A3. 19.2.1.  AFFTC  -  Surrogate  Carrier  Launch  Platform  (SCLP)  at  Utah  Test  and  Training  Range 
(UTTR)  Hill  AFB  UT.  A  modified  C-130  aircraft  for  externally  carrying  and  operating  develop¬ 
mental  cruise  missiles,  UAVs,  or  precision  munitions  (up  to  10,000  lbs  gross  weight)  for  captive 
flight  tests  or  developmental  launches.  Onboard  equipment  allows  receiving,  processing,  record¬ 
ing,  and  reradiating  of  test/TM  data  to  range  mission  control  centers.  Installed  airframe,  propul¬ 
sion,  and  avionic  subsystems  can  be  tested  and  evaluated  for  risk  reduction  prior  to  first  flights  of 
new  vehicles.  Contact:  545TG/XRP,  DSN:  458-7852. 

A3.20.  Propulsion: 


A3.21.  Avionics; 
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A3.21.1.1.  AFFTC  -  Advanced  Radar  Test  Bed  (ARTB).  A  modified/highly  instrumented  C-141. 
Various  airborne  fire  control  radar  (APG-63/70,  APG-66/68,  etc.)  hardware/software  can  be 
installed  (in  modified  radomes)  and  data  recorded  and/or  real-time  monitored  from  stations  in  rear 
of  aircraft.  Contact:  412  TW/OQ  DSN:  527-2410. 

A3.21.1.2.  AFDTC  -  Airborne  Seeker  Evaluation  Test  System  (ASETS).  A  modified/instru¬ 
mented  C-130  with  54  in.  turret  mounted  on  belly  of  aircraft  to  house  test  seekers  (IR,  millimeter 
wave,  laser,  etc.).  Instrumentation/data  recorders  are  in  racks  in  cargo-hold  of  aircraft.  Contact: 
46  TW/AFDTC,  DSN:  872-6414. 

A3.21.2.  CNI: 

A3.21.2.1.  Joint  Tactical  Information  Distribution  System  (JTIDS).  A  modified/instrumented 
T-39B  with  F-15  JTIDS  terminal  installed.  JTIDS  is  a  joint  service  program  to  design,  develop, 
secure,  jam-resistant,  digital  information  distribution  system  with  relative  navigation  and  positive 
identification  capabilities.  Contact:  AFDTC/XRC,  DSN:  872-2853. 

A3.21.2.2.  AFFTC  -  Satellite  Communication  (SATCOM).  A  modified/instrumented  C-135  for 
developing  and  testing  SATCOM  systems.  Modifications  included  addition  of  numerous  anten¬ 
nas,  25  in.  satellite  dish  under  tri-band  radome,  and  data/instrumentation  racks.  Contact:  452 
FLTS/DOOB  DSN:  527-8456. 

A3.21.2.3.  AFDTC  -  Completely  Integrated  Reference  Instrumentation  System  (CIRIS).  A  mod¬ 
ified/instrumented  C-141  which  contains  a  palletized  data  baseline  accurate  inertial  reference  sys¬ 
tem  for  flight  test  of  inertial  navigation  systems  (INS).  Contact:  46TW/AFDTC,  DSN:  787-3991. 

A3.21.3.  Support/Mission  Systems: 

A3.21.3.1.  AFFTC  -  Advanced  Fighter  Technology  Integrator  (AFTI/F-16),  Wright  Labs  but 
located  at  AFFTC,  Edwards  AFB,  Ca.  A  highly  modified  F-16  which  tests  assortments  of 
advanced  avionics/weapon  delivery  systems  including:  Ground  Collision  Avoidance,  Digital  Map 
Navigation,  and  Night  Attack  Systems  (advanced  sensors  and  helmet-mounted  sights/displays). 
Contact:  412  Test  Wing  (TW)/OGRN,  Phone:  (805)  258-3437. 

A3.21.3.2.  AFFTC  -  Avionic  Flying  Test  Bed  (AFTB).  A  highly  modified/instrumented  C-135A 
used  for  development  and  test  of  advanced  navigation,  radar,  and  avionic  integration.  Currently 
utilized  for  B-2  avionics/radar  development.  Modification  include  four  70/90  kilovolt  amp  (KVA) 
generators,  upgraded  cooling,  sensor  window,  GPS  antennas,  and  modified  radome.  Contact:  420 
FLTS/ENV,  DSN:  525-8993. 

A3.21.3.3.  AFFTC  -  Speckled  Trout.  Highly  modified  C-135C  to  test  advanced  cockpit  avionic 
systems.  Modifications  are  numerous  and  include:  GPS,  MIL-STD  1553  data  bus,  Barry 
Research  (BR)  communications  chirpsounder  [high  frequency  (HF)  tuner],  advanced  navigation 
systems  (Lasernav  II  and  Litton  92  systems),  and  flight  instrumentation  system.  Contact:  412 
FLTS/CC,DSN:  527-7791. 


